Xiyun Hu
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Xiyun Hu has a total of 30 co-authors. Here are the top co-authors by number of shared publications:
- Karthik Ramani
School of Mechanical Engineering, Purdue University, West Lafayette, Indiana, USA
12 shared publications - Ziyi Liu
School of Mechanical Engineering, Purdue University, West Lafayette, Indiana, USA
7 shared publications - Zhengzhe Zhu
Purdue University, West Lafayette, IN, USA
6 shared publications - Chenfei Zhu
School of Mechanical Engineering, Purdue University, West Lafayette, Indiana, USA
5 shared publications - Dizhi Ma
Elmore Family School of Electrical and Computer Engineering, Purdue University, West Lafayette, Indiana, USA
5 shared publications - Shao-Kang Hsia
Purdue University, West Lafayette, Indiana, USA
4 shared publications
Latest Publications
- Open Access
- 13 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-21
AgentCoach: LLM-Based Adaptive Coaching Feedback for Motor Skill Learning
- 13 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-21
Open AccessAgentCoach: LLM-Based Adaptive Coaching Feedback for Motor Skill Learning
- Open Access
- 13 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-31
JustShape: Exploring Co-Speech Gestures for Multimodal LLM-Powered 3D Parametric Modeling
- 13 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-31
Open AccessJustShape: Exploring Co-Speech Gestures for Multimodal LLM-Powered 3D Parametric Modeling
- Open Access
- 13 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-10
Towards Design Guidelines for Natural-Language-Based Human-Multi-Robot Collaboration in Domestic Environments
- 13 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-10
Open AccessTowards Design Guidelines for Natural-Language-Based Human-Multi-Robot Collaboration in Domestic Environments
- Open Access
- 13 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-23
ARify: Leveraging Narrated Instructional Videos to Create Augmented Reality Tutorials for Procedural Tasks
- 13 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-23
Open AccessARify: Leveraging Narrated Instructional Videos to Create Augmented Reality Tutorials for Procedural Tasks
- Open Access
- 13 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-28
AmIWrite: Exploring Scalable One-on-One Handwriting-Based Tutoring for Mathematical Problem-Solving with an LLM-Powered AI Tutor
- 13 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-28
Open AccessAmIWrite: Exploring Scalable One-on-One Handwriting-Based Tutoring for Mathematical Problem-Solving with an LLM-Powered AI Tutor
- Open Access
- 27 September 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-23
agentAR: Creating Augmented Reality Applications with Tool-Augmented LLM-based Autonomous Agents
- 27 September 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-23
Open AccessagentAR: Creating Augmented Reality Applications with Tool-Augmented LLM-based Autonomous Agents
- Top Cited PapersOpen Access
- 4 July 2025
- Published by Association for Computing Machinery (ACM)
- p. 59-80
GesPrompt: Leveraging Co-Speech Gestures to Augment LLM-Based Interaction in Virtual Reality
- 4 July 2025
- Published by Association for Computing Machinery (ACM)
- p. 59-80
Top Cited PapersOpen AccessGesPrompt: Leveraging Co-Speech Gestures to Augment LLM-Based Interaction in Virtual Reality
- Open Access
GesPrompt: Leveraging Co-Speech Gestures to Augment LLM-Based Interaction in Virtual Reality
Large Language Model (LLM)-based copilots have shown great potential in Extended Reality (XR) applications. However, the user faces challenges when describing the 3D environments to the copilots due to the complexity of conveying spatial-temporal information through text or speech alone. To address this, we introduce GesPrompt, a multimodal XR interface that combines co-speech gestures with speech, allowing end-users to communicate more naturally and accurately with LLM-based copilots in XR environments. By incorporating gestures, GesPrompt extracts spatial-temporal reference from co-speech gestures, reducing the need for precise textual prompts and minimizing cognitive load for end-users. Our contributions include (1) a workflow to integrate gesture and speech input in the XR environment, (2) a prototype VR system that implements the workflow, and (3) a user study demonstrating its effectiveness in improving user communication in VR environments.
Open AccessGesPrompt: Leveraging Co-Speech Gestures to Augment LLM-Based Interaction in Virtual Reality
Large Language Model (LLM)-based copilots have shown great potential in Extended Reality (XR) applications. However, the user faces challenges when describing the 3D environments to the copilots due to the complexity of conveying spatial-temporal information through text or speech alone. To address this, we introduce GesPrompt, a multimodal XR interface that combines co-speech gestures with speech, allowing end-users to communicate more naturally and accurately with LLM-based copilots in XR environments. By incorporating gestures, GesPrompt extracts spatial-temporal reference from co-speech gestures, reducing the need for precise textual prompts and minimizing cognitive load for end-users. Our contributions include (1) a workflow to integrate gesture and speech input in the XR environment, (2) a prototype VR system that implements the workflow, and (3) a user study demonstrating its effectiveness in improving user communication in VR environments.
- Open Access
- 24 October 2024
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Human-Computer Interaction
- Vol. 8 (ISS) , 44-69
AdapTUI: Adaptation of Geometric-Feature-Based Tangible User Interfaces in Augmented Reality
With the advents in geometry perception and Augmented Reality (AR), end-users can customize Tangible User Interfaces (TUIs) that control digital assets using intuitive and comfortable interactions with physical geometries (e.g., edges and surfaces). However, it remains challenging to adapt such TUIs in varied physical environments while maintaining the same spatial and ergonomic affordance. We propose AdapTUI, an end-to- end system that enables an end-user to author geometric-based TUIs and automatically adapts the TUIs when the user moves to a new environment. Leveraging a geometry detection module and the spatial awareness of AR, AdapTUI first lets users create custom mappings between geometric features and digital functions. Then, AdapTUI uses an optimization-based adaptation framework, which considers both the geometric variations and human-factor nuances, to dynamically adjust the attachment of the user-authored TUIs. We demonstrate three application scenarios where end-users can utilize TUIs at different locations, including portable car play, efficient AR workstation, and entertainment. We evaluated the effectiveness of the adaptation method as well as the overall usability through a comparison user study (N=12). The satisfactory adaptation of the user-authored TUIs and the positive qualitative feedback demonstrate the effectiveness of our system.
- 24 October 2024
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Human-Computer Interaction
- Vol. 8 (ISS) , 44-69
Open AccessAdapTUI: Adaptation of Geometric-Feature-Based Tangible User Interfaces in Augmented Reality
With the advents in geometry perception and Augmented Reality (AR), end-users can customize Tangible User Interfaces (TUIs) that control digital assets using intuitive and comfortable interactions with physical geometries (e.g., edges and surfaces). However, it remains challenging to adapt such TUIs in varied physical environments while maintaining the same spatial and ergonomic affordance. We propose AdapTUI, an end-to- end system that enables an end-user to author geometric-based TUIs and automatically adapts the TUIs when the user moves to a new environment. Leveraging a geometry detection module and the spatial awareness of AR, AdapTUI first lets users create custom mappings between geometric features and digital functions. Then, AdapTUI uses an optimization-based adaptation framework, which considers both the geometric variations and human-factor nuances, to dynamically adjust the attachment of the user-authored TUIs. We demonstrate three application scenarios where end-users can utilize TUIs at different locations, including portable car play, efficient AR workstation, and entertainment. We evaluated the effectiveness of the adaptation method as well as the overall usability through a comparison user study (N=12). The satisfactory adaptation of the user-authored TUIs and the positive qualitative feedback demonstrate the effectiveness of our system.
- Open Access
- 11 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-16
avaTTAR: Table Tennis Stroke Training with Embodied and Detached Visualization in Augmented Reality
Table tennis stroke training is a critical aspect of player development. We designed a new augmented reality (AR) system, avaTTAR, for table tennis stroke training. The system provides both “on-body” (first-person view) and “detached” (third-person view) visual cues, enabling users to visualize target strokes and correct their attempts effectively with this dual perspectives setup. By employing a combination of pose estimation algorithms and IMU sensors, avaTTAR captures and reconstructs the 3D body pose and paddle orientation of users during practice, allowing real-time comparison with expert strokes. Through a user study, we affirm avaTTAR ’s capacity to amplify player experience and training results.
- 11 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-16
Open AccessavaTTAR: Table Tennis Stroke Training with Embodied and Detached Visualization in Augmented Reality
Table tennis stroke training is a critical aspect of player development. We designed a new augmented reality (AR) system, avaTTAR, for table tennis stroke training. The system provides both “on-body” (first-person view) and “detached” (third-person view) visual cues, enabling users to visualize target strokes and correct their attempts effectively with this dual perspectives setup. By employing a combination of pose estimation algorithms and IMU sensors, avaTTAR captures and reconstructs the 3D body pose and paddle orientation of users during practice, allowing real-time comparison with expert strokes. Through a user study, we affirm avaTTAR ’s capacity to amplify player experience and training results.