Harnessing Memory Technology Innovations for On-Device Large Language Models in Augmented Reality SoCs: Cooptimizing system architecture with model characteristics
- 20 November 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Solid-State Circuits Magazine
- Vol. 17 (4) , 39-46
Abstract
Recent advancements in large language models (LLMs) have shown promise for human-like machine intelligence. Enabling these models in wearable augmented reality (AR) devices can elevate the user experiences with reduced latency and improved privacy, seamlessly blending advanced machine intelligence (artificial general intelligence) into daily life. However, deploying these models on wearable AR devices is hindered by the limited memory capacity and high energy/latency costs associated with off-chip memory access. This article assesses the recent memory technology innovations, including 3D-stacked memories, computing-in-memory (CIM), and processing-in-memory (PIM), and it explores how to utilize their potential to enable LLMs on AR SoCs. Using roofline modeling, we showcase the importance of cooptimizing system architecture with model characteristics to meet power/performance targets.Keywords
This publication has 17 references indexed in Scilit:
- DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learningNature, 2025
- Memory-Centric Computing: Solving Computing’s Memory Problem2025
- Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniques2025
- Mobile Edge Intelligence for Large Language Models: A Contemporary SurveyIEEE Communications Surveys & Tutorials, 2025
- Overcoming Memory Limitations for On-Device AI and LLM in Wearable AR Systems2024
- Generative AI Beyond LLMs: System Implications of Multi-Modal Generation2024
- System-Level Design and Integration of a Prototype AR/VR Hardware Featuring a Custom Low-Power DNN Accelerator Chip in 7nm Technology for Codec Avatars2022
- 3D V-Cache: the Implementation of a Hybrid-Bonded 64MB Stacked Cache for a 7nm x86-64 CPU2022
- A 5-nm 254-TOPS/W 221-TOPS/mm2 Fully-Digital Computing-in-Memory Macro Supporting Wide-Range Dynamic-Voltage-Frequency Scaling and Simultaneous MAC and Write Operations2022
- SoIC for Low-Temperature, Multi-Layer 3D Memory Integration2020