THTianyi Hu

Tianyi Hu

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Tianyi Hu has a total of 12 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    5G × Headsets: An Application-Level Measurement Study of 5G Latency for Networked Immersive XR

    by Tianyuan Du, Hanting Ye, Yiming Li, Tianyi Hu, Tingjun Chen, Maria Gorlatova

  • Open Access

    SHARE: Towards Head-Mounted AR with User-Centric SLAM in Shared Human-Robot Workspaces

    by Tianyuan Du, Tianyi Hu, Hanting Ye, Maria Gorlatova

    Human-Robot Collaboration (HRC) in shared physical spaces using Augmented Reality (AR) interfaces is powered by Simultaneous Localization and Mapping (SLAM). Existing multi-agent SLAM systems rely on an edge server to combine visual findings of multiple resource-constrained agents, perform computation, and schedule updates to their local maps. However, the edge treats all agents uniformly and ignores the fundamentally different latency requirements of heterogeneous HRC agents: robots and head-mounted AR users. This uniform resource allocation often results in high lag for user manipulation, as it does not meet the stringent latency requirements of AR. In this work, we design, implement, and evaluate SHARE, a user-centric SLAM system that strategically prioritizes AR user experience while maintaining accurate tracking performance for robots. SHARE builds a first-of-its-kind experience model for HRC agents and adaptively adjusts transmission priorities to match it. To reduce end-to-end latency, SHARE leverages the redundancy of visual features acquired by agents in shared human-robot workspaces to reduce computation time induced by edge-based processing. Real-world deployment with commercial AR headsets and a ground robot achieves 13.22 ms average latency for AR users (43.3% reduction from baseline) while maintaining sub-2-centimeter tracking accuracy. User studies further reveal statistically significant improvements in user perception.

  • Open Access

    Demo: Catch Smart Glasses If You Scan: Robot-Controlled vs. UI-Guided

    by Hanting Ye, Tianyi Hu, Valiantsin Kasabrukhau, Sulaiman Khydyr Uulu, Maria Gorlatova

    As smart glasses become increasingly lightweight and suitable for all-day wearables, they show strong potential to serve as the next-generation mobile platforms that provide an AR interface bridging human experience and AI perception. However, a serious concern about its misuse has recently been reported [2] and has attracted public attention: smart glasses can be used as cheating tools in competitions and exams when they are indistinguishable from regular glasses. Our previous work [3] introduces a non-invasive smart glasses detection solution that leverages two unique AR optical signatures found in current smart glasses: eye glow (i.e., light leakage from AR waveguide displays) and rainbow pattern (i.e., diffraction of ambient light on AR waveguide displays). In this approach, glasses need to be scanned manually using the cameras on off-the-shelf smartphones. However, it is challenging to maintain comprehensive, multi-angle coverage that guarantees consistent and accurate detection performance, as their scanning trajectories may vary significantly and are unable to capture frames containing eye glow or rainbow patterns. Inspired by this, in this demo, we introduce two glasses scanning pipelines: (1) a robot arm-based, controllable smart glasses scanning platform; and (2) a UI-guided smart glasses scanning solution. Pipeline Setup: In (1), the robot arm enables a reproducible smart glasses scanning process, where scanning trajectories can be customized by users. The scanning platform consists of a 4-DoF robot (PincherX 100 from Trossen Robotics), a 3D-printed face mask for mounting the smart glasses, and a smartphone camera on a stand, as shown in Figure 1(a). For robot control, we use a Linux-based Beelink Mini S12 running ROS2. We developed a UI with 11 predefined basic robot arm movements, which can be combined and adjusted in both order and amplitude of each movement to mimic human head motions within ranges of 140° yaw, 120° pitch, and 80° roll [1]. In (2), we develop a detector app on a Google Pixel 3 with UI indicators to guide users during scanning, ensuring that the smart glasses are observed from a comprehensive range of angles. We deploy the Google ML Kit face detection model to identify facial landmarks and use them to calculate the spatial orientation of the user's head pose. To simplify the user guidance experience, we create a ring around the user composed of 80 short grey bars, as shown in Figure 1(b). These bars represent different facial orientations. When a grey bar is covered for a certain period of time, it turns green. We also display instructional text at the bottom of the app to guide users to move toward any uncovered orientations until all the grey bars turn green. Interactive Demonstration: Participants can experience two smart glasses scanning pipelines. At the beginning, they can either use their own glasses or choose from the provided regular or smart glasses, with or without prescription lenses. In (1), participants place the selected glasses on the 3D-printed face mask and choose several basic head movements to combine, based on their own preferences for scanning behavior. The video recorded via the smartphone camera is sent to the edge for analysis to determine whether the selected scanning behavior can correctly distinguish smart glasses from regular ones. In (2), participants use our developed app to scan their face while wearing the glasses, following UI prompts until the ring indicators change from red to green, signaling that the scanning process is complete. The recorded video is then sent to the edge for analysis to determine whether the participant is wearing smart glasses. Through these two pipelines, demo participants can observe how an unoptimized scanning behavior, such as one produced by manually controlling the robot arm, might miss key AR optical signatures on smart glasses and result in detection failures, whereas a UI-guided design enables more comprehensive scanning and a higher detection success rate. Demonstration Video Link: https://youtu.be/6YOtyM3v9vw Acknowledgements. This work was supported in part by NSF grants CSR-2312760, CNS-2112562, and IIS-2231975, NSF CAREER Award IIS-2046072, NSF NAIAD Award 2332744, a Cisco Research Award, a Meta Research Award, Defense Advanced Research Projects Agency Young Faculty Award HR0011-24-1-0001, and the Army Research Laboratory under Cooperative Agreement Number W911NF-23-2-0224. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Defense Advanced Research Projects Agency, the Army Research Laboratory, or the U.S. Government. This paper has been approved for public release; distribution is unlimited. No official endorsement should be inferred. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.

  • Open Access

    No Guide, No Cheat: Detecting Smart Glasses via AR Optical Signatures

    by Hanting Ye, Tianyi Hu, Maria Gorlatova

    Smart glasses are gaining increasing attention as they are expected to complement smartphones as next-generation mobile platforms, providing an Augmented Reality (AR) interface that bridges human experience and Al perception. With the goal of making smart glasses suitable for daily use, a new security risk has emerged: smart glasses can be used as cheating tools in competitions and exams when they are indistinguishable from regular glasses. This paper delves into this issue by exploring AR display mechanisms: a waveguide embedded in the lens of smart glasses transmits light from the temples of the glasses to the user's eyes. Though the waveguide looks transparent from the front view, it induces AR displayed light leakage and unique light reflection patterns in a few other viewing angles. Motivated by this observation, we propose a new smart glasses detection method that exploits violations of the brightness constancy assumption in conventional optical flow methods to identify the unusual optical phenomena caused by the waveguide structure. Our preliminary experiments with three latest released smart glasses and three commonly seen regular glasses reveal that smart glasses can be detected in a non-invasive and privacy-preserving manner. This not only helps rebuild trustworthiness for smart glasses usage but also opens up new application opportunities that leverage AR unique optical signatures.


  • Open Access

    Apple v.s. Meta: A Comparative Study on Spatial Tracking in SOTA XR Headsets

    by Tianyi Hu, Fan Yang, Tim Scargill, Maria Gorlatova

    Inaccurate spatial tracking in extended reality (XR) headsets can cause virtual object jitter, misalignment, and user discomfort, limiting the headsets' potential for immersive content and natural interactions. We develop a modular testbed to evaluate the tracking performance of commercial XR headsets, incorporating system calibration, tracking data acquisition, and result analysis, and allowing the integration of external cameras and IMU sensors for comparison with open-source VI-SLAM algorithms. Using this testbed, we quantitatively assessed spatial tracking accuracy under various user movements and environmental conditions for the latest XR headsets, Apple Vision Pro and Meta Quest 3. The Apple Vision Pro outperformed the Meta Quest 3, reducing relative pose error (RPE) and absolute pose error (APE) by 33.9% and 14.6%, respectively. While both headsets achieved subcentimeter APE in most cases, they exhibited APE exceeding 10 cm in challenging scenarios, highlighting the need for further improvements in reliability and accuracy.


  • Open Access

    SEESys: Online Pose Error Estimation System for Visual SLAM

    by Tianyi Hu, Tim Scargill, Fan Yang, Ying Chen, Guohao Lan, Maria Gorlatova

    In this work, we introduce SEESys, the first system to provide online pose error estimation for Simultaneous Localization and Mapping (SLAM). Unlike prior offline error estimation approaches, the SEESys framework efficiently collects real-time system features and delivers accurate pose error magnitude estimates with low latency. This enables real-time quality-of-service information for downstream applications. To achieve this goal, we develop a SLAM system run-time status monitor (RTS monitor) that performs feature collection with minimal overhead, along with a multi-modality attention-based Deep SLAM Error Estimator (DeepSEE) for error estimation. We train and evaluate SEESys using both public SLAM benchmarks and a diverse set of synthetic datasets, achieving an RMSE of 0.235 cm of pose error estimation, which is 15.8% lower than the baseline. Additionally, we conduct a case study showcasing SEESys in a real-world scenario, where it is applied to a real-time audio error advisory system for human operators of a SLAM-enabled device. The results demonstrate that SEESys provides error estimates with an average end-to-end latency of 37.3 ms, and the audio error advisory reduces pose tracking error by 25%.


  • Open Access

    DNN-based SLAM Tracking Error Online Estimation

    by Tianyi Hu, Tim Scargill, Ying Chen, Guohao Lan, Maria Gorlatova

    Simultaneous localization and mapping (SLAM) takes in sensor data, e.g., camera frames, and estimates the user's trajectory while creating a map of the surrounding environment. However, existing SLAM evaluation methods are not reference-free, requiring ground-truth trajectories collected from external systems that are infeasible for most scenarios. In this demo, we present Deep SLAM Error Estimator (DeepSEE), a framework that collects features from a standard visual SLAM pipeline as multivariate time series and uses an attention-based neural network to estimate the tracking error at run time. We evaluate DeepSEE in a game engine-based virtual environment, which generates the visual input for DeepSEE and provides the ground-truth trajectory. Demo participants can navigate the virtual environment to create their own trajectories and view the online pose error estimation. This demo showcases how DeepSEE can act as a quality-of-service indicator for downstream applications.