Ziyun Li
Ziyun Li has a total of 93 co-authors. Here are the top co-authors by number of shared publications:
- Barbara De Salvo
Meta Reality Labs Research
8 shared publications - Syed Shakib Sarwar
Meta Reality Labs Research
7 shared publications - Jorge Gomez
Universidad de los Andes
6 shared publications - David Blaauw
Department of Electrical Engineering and Computer Science, University of Michigan 3 , Ann Arbor, Michigan 48109
5 shared publications - Hun-Seok Kim
University of Michigan
5 shared publications - Sai Qian Zhang
Tandon School of Engineering, New York University, New York, USA
4 shared publications
Latest Publications
- Open Access
- 10 April 2027
- Published by Association for Computing Machinery (ACM)
- p. 298-316
EDIVA: Exploiting Human Behavioral Patterns for Differential Egocentric Video Processing in AR
Augmented reality (AR) systems increasingly rely on real-time image segmentation to interpret continuous egocentric video streams and enable immersive, intention-aware interactions. However, executing neural segmentation models on every high-resolution frame introduces significant computational overhead and latency, exceeding the real-time constraints of AR devices. This paper presents EDIVA, an algorithm–hardware co-optimization framework for differential egocentric video processing that exploits temporal redundancy and multimodal sensory cues in AR environments. EDIVA uses gaze, pose, and inertial signals to infer user intent and reproject previous segmentation masks via perspective transformation, updating only regions with significant visual changes. A lightweight mask refinement module corrects residual distortions to maintain segmentation fidelity. We further design a dedicated EDIVA accelerator that integrates with the AR system-on-chip (SoC), enabling efficient cache management, mask reprojection, and reuse detection. Evaluations on representative AR workloads show that EDIVA reduces segmentation latency by 2.67× and energy consumption by 3.65× while preserving accuracy and temporal consistency, achieving real-time performance on edge AR hardware. Code is available at https://doi.org/10.5281/zenodo.23225199.
- 10 April 2027
- Published by Association for Computing Machinery (ACM)
- p. 298-316
Open AccessEDIVA: Exploiting Human Behavioral Patterns for Differential Egocentric Video Processing in AR
Augmented reality (AR) systems increasingly rely on real-time image segmentation to interpret continuous egocentric video streams and enable immersive, intention-aware interactions. However, executing neural segmentation models on every high-resolution frame introduces significant computational overhead and latency, exceeding the real-time constraints of AR devices. This paper presents EDIVA, an algorithm–hardware co-optimization framework for differential egocentric video processing that exploits temporal redundancy and multimodal sensory cues in AR environments. EDIVA uses gaze, pose, and inertial signals to infer user intent and reproject previous segmentation masks via perspective transformation, updating only regions with significant visual changes. A lightweight mask refinement module corrects residual distortions to maintain segmentation fidelity. We further design a dedicated EDIVA accelerator that integrates with the AR system-on-chip (SoC), enabling efficient cache management, mask reprojection, and reuse detection. Evaluations on representative AR workloads show that EDIVA reduces segmentation latency by 2.67× and energy consumption by 3.65× while preserving accuracy and temporal consistency, achieving real-time performance on edge AR hardware. Code is available at https://doi.org/10.5281/zenodo.23225199.
- 15 September 2026
- Published by Springer Nature
- p. 144-162
EgoEverything: A Benchmark for Human Behavior–Inspired Long-Context Egocentric Video Understanding in AR Environment
- 15 September 2026
- Published by Springer Nature
- p. 144-162
EgoEverything: A Benchmark for Human Behavior–Inspired Long-Context Egocentric Video Understanding in AR Environment
- Open Access
- 28 July 2025
- Published by Association for Computing Machinery (ACM)
- p. 28-30
H4H: Hybrid Convolution-Transformer Architecture Search for NPU-CIM Heterogeneous Systems for AR/VR Applications (Abstract)
- 28 July 2025
- Published by Association for Computing Machinery (ACM)
- p. 28-30
Open AccessH4H: Hybrid Convolution-Transformer Architecture Search for NPU-CIM Heterogeneous Systems for AR/VR Applications (Abstract)
- Open Access
- 20 January 2025
- Published by Association for Computing Machinery (ACM)
- p. 1133-1141
H4H: Hybrid Convolution-Transformer Architecture Search for NPU-CIM Heterogeneous Systems for AR/VR Applications
Low-latency and low-power edge AI is crucial for Augmented/Virtual Reality applications. Recent advances demonstrate that hybrid models, combining convolution layers (CNN) and transformers (ViT), often achieve a superior accuracy/performance tradeoff on various computer vision and machine learning (ML) tasks. However, hybrid ML models can present system challenges for latency and energy efficiency due to their diverse nature in dataflow and memory access patterns. In this work, we leverage architecture heterogeneity from Neural Processing Units (NPU) and Compute-In-Memory (CIM) and explore diverse execution schemas for efficient hybrid model executions. We introduce H4H-NAS, a two-stage Neural Architecture Search (NAS) framework to automate the design of hybrid CNN/ViT models for heterogeneous edge systems featuring both NPU and CIM. We propose a two-phase incremental supernet training in our NAS to resolve gradient conflicts between sampled subnets caused by different block types in a hybrid model search space. Our H4H-NAS approach is also powered by a performance estimator built with NPU performance results measured on real silicon, and CIM performance based on industry IPs. H4H-NAS searches hybrid CNN-ViT models with fine granularity and achieves significant (up to 1.34%) top-1 accuracy improvement on ImageNet-1k. Moreover, results from our algorithm/hardware co-design reveal up to 56.08% overall latency and 41.72% energy improvements by introducing heterogeneous computing over baseline solutions. Overall, our framework guides the design of hybrid network architectures and system architectures for NPU+CIM heterogeneous systems.
- 20 January 2025
- Published by Association for Computing Machinery (ACM)
- p. 1133-1141
Open AccessH4H: Hybrid Convolution-Transformer Architecture Search for NPU-CIM Heterogeneous Systems for AR/VR Applications
Low-latency and low-power edge AI is crucial for Augmented/Virtual Reality applications. Recent advances demonstrate that hybrid models, combining convolution layers (CNN) and transformers (ViT), often achieve a superior accuracy/performance tradeoff on various computer vision and machine learning (ML) tasks. However, hybrid ML models can present system challenges for latency and energy efficiency due to their diverse nature in dataflow and memory access patterns. In this work, we leverage architecture heterogeneity from Neural Processing Units (NPU) and Compute-In-Memory (CIM) and explore diverse execution schemas for efficient hybrid model executions. We introduce H4H-NAS, a two-stage Neural Architecture Search (NAS) framework to automate the design of hybrid CNN/ViT models for heterogeneous edge systems featuring both NPU and CIM. We propose a two-phase incremental supernet training in our NAS to resolve gradient conflicts between sampled subnets caused by different block types in a hybrid model search space. Our H4H-NAS approach is also powered by a performance estimator built with NPU performance results measured on real silicon, and CIM performance based on industry IPs. H4H-NAS searches hybrid CNN-ViT models with fine granularity and achieves significant (up to 1.34%) top-1 accuracy improvement on ImageNet-1k. Moreover, results from our algorithm/hardware co-design reveal up to 56.08% overall latency and 41.72% energy improvements by introducing heterogeneous computing over baseline solutions. Overall, our framework guides the design of hybrid network architectures and system architectures for NPU+CIM heterogeneous systems.
- Open Access
- 18 September 2024
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Design Automation of Electronic Systems
- Vol. 29 (6) , 1-27
Estimating Power, Performance, and Area for On-Sensor Deployment of AR/VR Workloads Using an Analytical Framework
Augmented Reality and Virtual Reality have emerged as the next frontier of intelligent image sensors and computer systems. In these systems, 3D die stacking stands out as a compelling solution, enabling in situ processing capability of the sensory data for tasks such as image classification and object detection at low power, low latency, and a small form factor. These intelligent 3D CMOS Image Sensor (CIS) systems present a wide design space, encompassing multiple domains (e.g., computer vision algorithms, circuit design, system architecture, and semiconductor technology, including 3D stacking) that have not been explored in-depth so far. This article aims to fill this gap. We first present an analytical evaluation framework, STAR-3DSim, dedicated to rapid pre-RTL evaluation of 3D-CIS systems capturing the entire stack from the pixel layer to the on-sensor processor layer. With STAR-3DSim, we then propose several knobs for PPA (power, performance, area) improvement of the Deep Neural Network (DNN) accelerator that can provide up to 53%, 41%, and 63% reduction in energy, latency, and area, respectively, across a broad set of relevant AR/VR workloads. Last, we present full-system evaluation results by taking image sensing, cross-tier data transfer, and off-sensor communication into consideration.
- 18 September 2024
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Design Automation of Electronic Systems
- Vol. 29 (6) , 1-27
Open AccessEstimating Power, Performance, and Area for On-Sensor Deployment of AR/VR Workloads Using an Analytical Framework
Augmented Reality and Virtual Reality have emerged as the next frontier of intelligent image sensors and computer systems. In these systems, 3D die stacking stands out as a compelling solution, enabling in situ processing capability of the sensory data for tasks such as image classification and object detection at low power, low latency, and a small form factor. These intelligent 3D CMOS Image Sensor (CIS) systems present a wide design space, encompassing multiple domains (e.g., computer vision algorithms, circuit design, system architecture, and semiconductor technology, including 3D stacking) that have not been explored in-depth so far. This article aims to fill this gap. We first present an analytical evaluation framework, STAR-3DSim, dedicated to rapid pre-RTL evaluation of 3D-CIS systems capturing the entire stack from the pixel layer to the on-sensor processor layer. With STAR-3DSim, we then propose several knobs for PPA (power, performance, area) improvement of the Deep Neural Network (DNN) accelerator that can provide up to 53%, 41%, and 63% reduction in energy, latency, and area, respectively, across a broad set of relevant AR/VR workloads. Last, we present full-system evaluation results by taking image sensing, cross-tier data transfer, and off-sensor communication into consideration.
- 28 June 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Very Large Scale Integration (VLSI) Systems
- Vol. 32 (9) , 1718-1725
Thermally Constrained Codesign of Heterogeneous 3-D Integration of Compute-in-Memory, Digital ML Accelerator, and RISC-V Cores for Mixed ML and Non-ML Workloads
Heterogeneous 3-D (H3D) integration not only reduces the chip form factor and fabrication cost but also allows the merging of diverse compute paradigms that suit different applications. This is especially attractive when modern algorithms, such as the augmented reality/virtual reality (AR/VR) workloads, consist of mixed machine learning (ML) and non-ML workloads. To date, codesign that considers the thermal, latency, and power constraints of H3D hardware is largely unexplored. In this work, a thermally aware framework for H3D hardware design is developed to evaluate the thermal, latency, and power trade-offs for a heterogeneous system with compute-in-memory (CIM), digital ML cores, and RISC-V cores. The framework solves for runtime tunable operating points described as the optimal speedup factor, the number of activated RISC-V cores, the cooling coefficient, and the activity rate based on user-defined criteria, achieving up to 135 TOPS and 215 TOPS/W under $74~^{\circ }$ C for the AR/VR workloads.
- 28 June 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Very Large Scale Integration (VLSI) Systems
- Vol. 32 (9) , 1718-1725
Thermally Constrained Codesign of Heterogeneous 3-D Integration of Compute-in-Memory, Digital ML Accelerator, and RISC-V Cores for Mixed ML and Non-ML Workloads
Heterogeneous 3-D (H3D) integration not only reduces the chip form factor and fabrication cost but also allows the merging of diverse compute paradigms that suit different applications. This is especially attractive when modern algorithms, such as the augmented reality/virtual reality (AR/VR) workloads, consist of mixed machine learning (ML) and non-ML workloads. To date, codesign that considers the thermal, latency, and power constraints of H3D hardware is largely unexplored. In this work, a thermally aware framework for H3D hardware design is developed to evaluate the thermal, latency, and power trade-offs for a heterogeneous system with compute-in-memory (CIM), digital ML cores, and RISC-V cores. The framework solves for runtime tunable operating points described as the optimal speedup factor, the number of activated RISC-V cores, the cooling coefficient, and the activity rate based on user-defined criteria, achieving up to 135 TOPS and 215 TOPS/W under $74~^{\circ }$ C for the AR/VR workloads.
- 23 May 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Solid-State Circuits
- Vol. 59 (7) , 2055-2069
Siracusa: A 16 nm Heterogenous RISC-V SoC for Extended Reality With At-MRAM Neural Engine
Extended reality (XR) applications are machine learning (ML)-intensive, featuring deep neural networks (DNNs) with millions of weights, tightly latency-bound (10–20 ms end-to-end), and power-constrained (low tens of mW average power). While ML performance and efficiency can be achieved by introducing neural engines within low-power systems-on-chip (SoCs), system-level power for nontrivial DNNs depends strongly on the energy of non-volatile memory (NVM) access for network weights. This work introduces Siracusa , a near-sensor heterogeneous SoC for next-generation XR devices manufactured in 16 nm CMOS. Siracusa couples an octa-core cluster of RISC-V digital signal processing (DSP) cores with a novel tightly coupled “At-Memory” integration between a state-of-the-art digital neural engine called and an on-chip NVM based on magnetoresistive random access memory (MRAM), achieving 1.7 $\times$ higher throughput and 3 $\times$ better energy efficiency than XR SoCs using NVM as background memory. The fabricated SoC prototype achieves an area efficiency of 65.2 GOp/s/mm ${}^\text{2}$ and a peak energy efficiency of 8.84 TOp/J for DNN inference while supporting complex, heterogeneous application workloads, which combine ML with conventional signal processing and control.
- 23 May 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Solid-State Circuits
- Vol. 59 (7) , 2055-2069
Siracusa: A 16 nm Heterogenous RISC-V SoC for Extended Reality With At-MRAM Neural Engine
Extended reality (XR) applications are machine learning (ML)-intensive, featuring deep neural networks (DNNs) with millions of weights, tightly latency-bound (10–20 ms end-to-end), and power-constrained (low tens of mW average power). While ML performance and efficiency can be achieved by introducing neural engines within low-power systems-on-chip (SoCs), system-level power for nontrivial DNNs depends strongly on the energy of non-volatile memory (NVM) access for network weights. This work introduces Siracusa , a near-sensor heterogeneous SoC for next-generation XR devices manufactured in 16 nm CMOS. Siracusa couples an octa-core cluster of RISC-V digital signal processing (DSP) cores with a novel tightly coupled “At-Memory” integration between a state-of-the-art digital neural engine called and an on-chip NVM based on magnetoresistive random access memory (MRAM), achieving 1.7 $\times$ higher throughput and 3 $\times$ better energy efficiency than XR SoCs using NVM as background memory. The fabricated SoC prototype achieves an area efficiency of 65.2 GOp/s/mm ${}^\text{2}$ and a peak energy efficiency of 8.84 TOp/J for DNN inference while supporting complex, heterogeneous application workloads, which combine ML with conventional signal processing and control.
- Open Access
Siracusa: A 16 nm Heterogenous RISC-V SoC for Extended Reality with At-MRAM Neural Engine
Extended reality (XR) applications are Machine Learning (ML)-intensive, featuring deep neural networks (DNNs) with millions of weights, tightly latency-bound (10-20 ms end-to-end), and power-constrained (low tens of mW average power). While ML performance and efficiency can be achieved by introducing neural engines within low-power systems-on-chip (SoCs), system-level power for nontrivial DNNs depends strongly on the energy of non-volatile memory (NVM) access for network weights. This work introduces Siracusa, a near-sensor heterogeneous SoC for next-generation XR devices manufactured in 16 nm CMOS. Siracusa couples an octa-core cluster of RISC-V digital signal processing cores with a novel tightly-coupled "At-Memory" integration between a state-of-the-art digital neural engine called N-EUREKA and an on-chip NVM based on magnetoresistive memory(MRAM), achieving 1.7x higher throughput and 3x better energy efficiency than XR SoCs using NVM as background memory. The fabricated SoC prototype achieves an area efficiency of 65.2 GOp/s/mm2 and a peak energy efficiency of 8.84 TOp/J for DNN inference while supporting complex heterogeneous application workloads, which combine ML with conventional signal processing and control.
Open AccessSiracusa: A 16 nm Heterogenous RISC-V SoC for Extended Reality with At-MRAM Neural Engine
Extended reality (XR) applications are Machine Learning (ML)-intensive, featuring deep neural networks (DNNs) with millions of weights, tightly latency-bound (10-20 ms end-to-end), and power-constrained (low tens of mW average power). While ML performance and efficiency can be achieved by introducing neural engines within low-power systems-on-chip (SoCs), system-level power for nontrivial DNNs depends strongly on the energy of non-volatile memory (NVM) access for network weights. This work introduces Siracusa, a near-sensor heterogeneous SoC for next-generation XR devices manufactured in 16 nm CMOS. Siracusa couples an octa-core cluster of RISC-V digital signal processing cores with a novel tightly-coupled "At-Memory" integration between a state-of-the-art digital neural engine called N-EUREKA and an on-chip NVM based on magnetoresistive memory(MRAM), achieving 1.7x higher throughput and 3x better energy efficiency than XR SoCs using NVM as background memory. The fabricated SoC prototype achieves an area efficiency of 65.2 GOp/s/mm2 and a peak energy efficiency of 8.84 TOp/J for DNN inference while supporting complex heterogeneous application workloads, which combine ML with conventional signal processing and control.
- 27 February 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Solid-State Circuits
- Vol. 59 (8) , 2644-2658
RoboVisio: A Micro-Robot Vision Domain-Specific SoC for Autonomous Navigation Enabling Fully-on-Chip Intelligence via 2-MB eMRAM
This article presents RoboVisio, an efficient and highly flexible domain-specific system-on-chip (SoC) for vision tasks in fully autonomous micro-robot navigation. A novel hybrid processing element (PE) is proposed, in which classic vision tasks achieve high efficiency by using a 2-D-mapping architecture, while convolutional neural network (CNN) is executed in an efficient output-channel-parallel systolic manner. Combining both processing schemes into a single PE array future-proofs the architecture, facilitating next-generation CNN-heavy vision algorithms, while saving 40% area and leakage with no power overhead and throughput loss, compared with two separate array implementations. To further improve energy and area efficiency, the design incorporates a number of key features: 1) 2-MB magnetoresistive random access memory (MRAM) for non-volatile fully-on-chip weight storage; 2) a unified image-activation memory (IAMEM) with block-swapping-based input/output image buffering that reduces buffer footprint by 50% and eliminates data copy for multi-frame buffering; and 3) a combination of weight buffering and CNN loop ordering that reduces weight memory system power by 75%. Fabricated in 22-nm CMOS, the design achieves 0.22 nJ/pixels for Harris corner feature detection (a classic or non-CNN vision task) and 3.5 TOPS/W (16-bit OP) for CNN, a 40%–170% efficiency improvement over state-of-the-art edge machine learning (ML) SoCs using non-volatile memory (NVM).
- 27 February 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Solid-State Circuits
- Vol. 59 (8) , 2644-2658
RoboVisio: A Micro-Robot Vision Domain-Specific SoC for Autonomous Navigation Enabling Fully-on-Chip Intelligence via 2-MB eMRAM
This article presents RoboVisio, an efficient and highly flexible domain-specific system-on-chip (SoC) for vision tasks in fully autonomous micro-robot navigation. A novel hybrid processing element (PE) is proposed, in which classic vision tasks achieve high efficiency by using a 2-D-mapping architecture, while convolutional neural network (CNN) is executed in an efficient output-channel-parallel systolic manner. Combining both processing schemes into a single PE array future-proofs the architecture, facilitating next-generation CNN-heavy vision algorithms, while saving 40% area and leakage with no power overhead and throughput loss, compared with two separate array implementations. To further improve energy and area efficiency, the design incorporates a number of key features: 1) 2-MB magnetoresistive random access memory (MRAM) for non-volatile fully-on-chip weight storage; 2) a unified image-activation memory (IAMEM) with block-swapping-based input/output image buffering that reduces buffer footprint by 50% and eliminates data copy for multi-frame buffering; and 3) a combination of weight buffering and CNN loop ordering that reduces weight memory system power by 75%. Fabricated in 22-nm CMOS, the design achieves 0.22 nJ/pixels for Harris corner feature detection (a classic or non-CNN vision task) and 3.5 TOPS/W (16-bit OP) for CNN, a 40%–170% efficiency improvement over state-of-the-art edge machine learning (ML) SoCs using non-volatile memory (NVM).
- Open Access
- 28 December 2023
- Published by Mason Publish Group in Electrical & Electronic Engineering Research
- Vol. 3 (1)
A Fast Fixed-Point Logarithmic Functions Based on ARM Microprocessor
The fixed-point mathematical library refers to a highly optimized collection of high-precision mathematical functions primarily used for fast and high-precision real-time calculations. It can execute equivalent code written in the C language with floating-point format faster, while maintaining considerable accuracy. Current mainstream fixed-point mathematical libraries face issues such as being non-open source, having unknown models, incomplete basic mathematical operation functions, and insufficient precision. Therefore, designing an open-source, fast, more optimized, and high-precision fixed-point mathematical library will have a groundbreaking impact. It can not only play a crucial role in industrial control algorithms but also eliminate the dependency of domestic fixed-point MCUs on foreign library functions. This paper conducts research on fixed-point arithmetic for logarithmic functions, designs, and improves the implementation of fast logarithmic functions.Taking Q12 as an example, Simulation experiments and MCU experiments show that, in the Q12 format, the highest precision that can be represented by a 32-bit fixed-point number is 0.000244141. From the test results, it can be observed that for 98.15% of the test data, the computational errors of the two mathematical libraries are both smaller than the highest precision value. Both two logarithmic function computations can maintain good computational accuracy within the ARM microprocessor. In terms of computational speeds, the average computation cycle of the fixed-point logarithmic function designed in this paper is reduced by 31.88% compared to the counterpart. This substantial improvement in computational speed, while ensuring computational accuracy, enhances the performance of fixed-point logarithmic operations based on ARM microprocessors.
- 28 December 2023
- Published by Mason Publish Group in Electrical & Electronic Engineering Research
- Vol. 3 (1)
Open AccessA Fast Fixed-Point Logarithmic Functions Based on ARM Microprocessor
The fixed-point mathematical library refers to a highly optimized collection of high-precision mathematical functions primarily used for fast and high-precision real-time calculations. It can execute equivalent code written in the C language with floating-point format faster, while maintaining considerable accuracy. Current mainstream fixed-point mathematical libraries face issues such as being non-open source, having unknown models, incomplete basic mathematical operation functions, and insufficient precision. Therefore, designing an open-source, fast, more optimized, and high-precision fixed-point mathematical library will have a groundbreaking impact. It can not only play a crucial role in industrial control algorithms but also eliminate the dependency of domestic fixed-point MCUs on foreign library functions. This paper conducts research on fixed-point arithmetic for logarithmic functions, designs, and improves the implementation of fast logarithmic functions.Taking Q12 as an example, Simulation experiments and MCU experiments show that, in the Q12 format, the highest precision that can be represented by a 32-bit fixed-point number is 0.000244141. From the test results, it can be observed that for 98.15% of the test data, the computational errors of the two mathematical libraries are both smaller than the highest precision value. Both two logarithmic function computations can maintain good computational accuracy within the ARM microprocessor. In terms of computational speeds, the average computation cycle of the fixed-point logarithmic function designed in this paper is reduced by 31.88% compared to the counterpart. This substantial improvement in computational speed, while ensuring computational accuracy, enhances the performance of fixed-point logarithmic operations based on ARM microprocessors.