Edith Beigne

Recent advancements in large language models (LLMs) have shown promise for human-like machine intelligence. Enabling these models in wearable augmented reality (AR) devices can elevate the user experiences with reduced latency and improved privacy, seamlessly blending advanced machine intelligence (artificial general intelligence) into daily life. However, deploying these models on wearable AR devices is hindered by the limited memory capacity and high energy/latency costs associated with off-chip memory access. This article assesses the recent memory technology innovations, including 3D-stacked memories, computing-in-memory (CIM), and processing-in-memory (PIM), and it explores how to utilize their potential to enable LLMs on AR SoCs. Using roofline modeling, we showcase the importance of cooptimizing system architecture with model characteristics to meet power/performance targets.

Compute-in-memory (CIM) has emerged as an effective technique to address memory access bottleneck for Deep Neural Networks (DNN). Augmented/Virtual Reality (AR/VR) devices require running high-performance DNN inference at tight power budgets, making CIMs ideal candidates for low-power on-device acceleration. While high energy-efficiencies have been reported at the CIM macro levels, system-on-chip (SoC) level energy-efficiency of CIM based accelerators have been underexplored for realistic system integration considerations. In this work, we present a CIM-accelerator architecture comprised of 16 row-pipelined fully-digital CIM macros, and provide a comprehensive analysis of CIM energy-efficiency benefits at the SoC-level targeting representative AR/VR workloads. Two key results are: (1) Realistic SoC-level CIM-accelerator energy-efficiency may be ∼50% lower than the CIM macro-level peak energy-efficiency when additional logic, memory hierarchies, and NN-dependent sub-optimal compute utilization are considered. (2) The CIM-accelerator still demonstrates up to ∼2.1× energy savings at the SoC-level compared to a systolic-array-based DNN accelerator at iso-peak throughput.

Three-dimensional integration offers architectural and performance benefits for scaling augmented/virtual reality (AR/VR) models on highly resource-constrained edge devices. Two-dimensional off-chip memory interfaces are too prohibitively energy intensive and bandwidth (BW) limited for AR/VR devices. To solve this, we propose using advanced 3-D stacking technology for high-density vertical integration to local memory and compute, increasing memory capacity within the same footprint at iso-BW with improvements in energy and latency. We evaluate 3-D architectures for a prototype AR/VR accelerator to demonstrate up to 3.9× latency reduction and 1.6× lower energy compared to a 2-D configuration within a smaller/similar footprint. Additionally, we show the feasibility of deploying higher resolution AR/VR models by stacking multiple tiers of memory, providing a pathway to break the footprint constraints of 2-D architectures. The use of high-density 3-D interconnects allows us to demonstrate localized benefits at the accelerator-level compared with standard system-on-chip memory disaggregation techniques/architectures.

Augmented Reality / Virtual Reality (AR/VR) devices aim to connect people in the Metaverse with photorealistic virtual avatars, referred to as “Codec Avatars”. Delivering a high visual performance for Codec Avatar workloads, however, is a challenging task for mobile SoCs as AR/VR devices have limited power and form factor constraints. On-device, local, near-sensor processing provides the best system-level energy-efficiency and enables strong security and privacy features in the long run. In this work, we present a custom-built, prototype small-scale mobile SoC that achieves energy-efficient performance for running eye gaze extraction of the Codec Avatar model. The test-chip, fabricated in 7nm technology node, features a Neural Network (NN) accelerator consisting of a 1024 Multiply-Accumulate (MAC) array, 2MB on-chip SRAM, and a 32bit RISC-V CPU. The featured test-chip is integrated on a prototype mobile VR headset to run the Codec Avatar application. This work aims to show the full stack design considerations of system-level integration, hardware-aware model customization, and circuit-level acceleration to meet the challenging mobile AR/VR SoC specifications for a Codec Avatar demonstration. By re-architecting the Convolutional NN (CNN) based eye gaze extraction model and tailoring it for the hardware, the entire model fits on the chip to mitigate system-level energy and latency cost of off-chip memory accesses. By efficiently accelerating the convolution operation at the circuit-level, the presented prototype SoC achieves 30 frames per second performance with low-power consumption at low form factors. With the full-stack design considerations presented in this work, the featured test-chip consumes 22.7mW power to run inference on the entire CNN model in 16.5ms from input to output for a single sensor image. As a result, the test-chip achieves 375 µJ/frame/eye energy-efficiency within a 2.56 mm ² silicon area.

This paper presents a single-rail two-port static random-access memory (SRAM) designed in 28 nm FD-SOI technology specifically for a synchronous/asynchronous IoT node. This SRAM supports an asynchronous interface communication and a fast transition sleep/active mode. It enables simultaneous synchronous and asynchronous accesses on its two independent ports, as well as a selective virtual ground (SVGND) to support ultra-low voltage (ULV) in read operations. Measurements on a 64 kbits memory macro prove the functionality for a supply voltage ranging from 0.25 V to 1.25 V. The sleep mode enables 158x reduction of the SVGND static-power consumption. The SVGND read assist achieves a gain of 50 mV on the read port operations lowest voltage. The memory achieves, at 0.25 V and 27∘C, an average energy cost per access of 1.45 fJ/bit at 451 kHz and a leakage of 25 pW/bit in sleep mode.

Due to low activity in Internet of Things (IoT) applications, systems tend to leverage low power modes in order to reduce their power consumption. Normally-off computing thus arose, consisting in having turned off most part of a system’s power supply, while dynamically turning on components as the application needs it. As wake up sources may be diverse, simple controllers are integrated to handle smart wake up schemes. Therefore, to prevent overconsumption while transitioning to running mode, fast wake up sequences are required. An asynchronous 16-bit Reduced Instruction Set Computer (RISC) Wake-up Controller (WuC) is proposed demonstrating 50.5 ns@9.2 Million Instructions Per Second (MIPS)@0.6 V wake-up latency, drastically reducing the overall wake-up energy of IoT systems. A clockless implementation of the controller saves the booting time and the power consumption of a clock generator, while providing high robustness to environmental variations such as supply voltage level. The WuC is also able to run simple tasks with a reduced Instruction Set Architecture (ISA) and achieves as low as 11.2 pJ/inst @0.5 V in Fully Depleted Silicon On Insulator (FDSOI) 28 nm.

This paper presents a 3-D stacked vision chip featuring in-focal-plane read-out tightly coupled with flexible computing architecture for configurable high-speed image analysis. The chip architecture is based on a scalable standalone structure integrating image sensor on the top tier and processing elements (PEs) plus memories in the bottom tier. By using 3-D stacking partitioning, our prototype benefits from backside illuminated pixels sensitivity, a fully parallel communication between image sensor and PEs for low-latency performances, while leaving enough room in the bottom tier to embed advanced computing features. One scalable structure embeds a 16x16 pixel array (or 64x64 pixels in high-resolution mode), associated with an 8-bit single instruction multiple data (SIMD) processor; fabricated in dual 130-nm 1P6M CMOS process. This paper exhibits a 5500 frames/s and 85 giga operations per second (GOPS)/W in low-resolution mode, with large kernels capabilities through eight directions interpixel communication. Multiflow capability is also demonstrated to execute different programs in different areas of the vision chip.