Jingyu Shi
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Jingyu Shi has a total of 30 co-authors. Here are the top co-authors by number of shared publications:
- Karthik Ramani
School of Mechanical Engineering, Purdue University, West Lafayette, Indiana, USA
11 shared publications - Rahul Jain
Department of Electrical and Computer Engineering, Purdue University, West Lafayette, Indiana, USA
7 shared publications - Chenfei Zhu
School of Mechanical Engineering, Purdue University, West Lafayette, Indiana, USA
4 shared publications - Runlin Duan
School of Mechanical Engineering, Purdue University, West Lafayette, Indiana, USA
4 shared publications - Xiyun Hu
School of Mechanical Engineering, Purdue University, West Lafayette, Indiana, USA
3 shared publications - Yichen Hu
Department of Computer Sciences, Purdue University, West Lafayette, Indiana, USA
3 shared publications
Latest Publications
- Open Access
- 12 June 2026
- Published by Association for Computing Machinery (ACM)
- p. 605-620
SketchConcept: Sketching-based Concept Composition for Product Design using Multimodal Large Language Model
- 12 June 2026
- Published by Association for Computing Machinery (ACM)
- p. 605-620
Open AccessSketchConcept: Sketching-based Concept Composition for Product Design using Multimodal Large Language Model
- Open Access
- 22 March 2026
- Published by Association for Computing Machinery (ACM)
- p. 1362-1386
Canvas3D: Empowering Precise Spatial Control for Image Generation with Constraints from a 3D Virtual Canvas
Generative AI (GenAI) has significantly advanced the ease and flexibility of image creation. However, it remains a challenge to precisely control spatial compositions, including object arrangement and scene conditions. To bridge this gap, we propose Canvas3D, an interactive system leveraging a 3D engine to enable precise spatial manipulation for image generation. Upon user prompt, Canvas3D automatically converts textual descriptions into interactive objects within a 3D engine-driven virtual canvas, empowering direct and precise spatial configuration. These user-defined arrangements generate explicit spatial constraints that guide generative models in accurately reflecting user intentions in the resulting images. We conducted a closed-ended comparative study between Canvas3D and a baseline system, and an open-ended, free-form study to assess overall system usability. The results indicate that Canvas3D outperforms the baseline on spatial control, interactivity, and overall user experience.
- 22 March 2026
- Published by Association for Computing Machinery (ACM)
- p. 1362-1386
Open AccessCanvas3D: Empowering Precise Spatial Control for Image Generation with Constraints from a 3D Virtual Canvas
Generative AI (GenAI) has significantly advanced the ease and flexibility of image creation. However, it remains a challenge to precisely control spatial compositions, including object arrangement and scene conditions. To bridge this gap, we propose Canvas3D, an interactive system leveraging a 3D engine to enable precise spatial manipulation for image generation. Upon user prompt, Canvas3D automatically converts textual descriptions into interactive objects within a 3D engine-driven virtual canvas, empowering direct and precise spatial configuration. These user-defined arrangements generate explicit spatial constraints that guide generative models in accurately reflecting user intentions in the resulting images. We conducted a closed-ended comparative study between Canvas3D and a baseline system, and an open-ended, free-form study to assess overall system usability. The results indicate that Canvas3D outperforms the baseline on spatial control, interactivity, and overall user experience.
- Open Access
- 27 September 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-23
agentAR: Creating Augmented Reality Applications with Tool-Augmented LLM-based Autonomous Agents
- 27 September 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-23
Open AccessagentAR: Creating Augmented Reality Applications with Tool-Augmented LLM-based Autonomous Agents
- Open Access
- 4 July 2025
- Published by Association for Computing Machinery (ACM)
- p. 1040-1060
DesignFromX:Empowering Consumer-Driven Design Space Exploration through Feature Composition of Referenced Products
- 4 July 2025
- Published by Association for Computing Machinery (ACM)
- p. 1040-1060
Open AccessDesignFromX:Empowering Consumer-Driven Design Space Exploration through Feature Composition of Referenced Products
- Open Access
- 6 May 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-3
Generalized TensoRF: Efficient Multi-Scene Radiance Fields and View-Consistent 3D Editing
- 6 May 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-3
Open AccessGeneralized TensoRF: Efficient Multi-Scene Radiance Fields and View-Consistent 3D Editing
- Open Access
- 25 April 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-23
CARING-AI: Towards Authoring Context-aware Augmented Reality INstruction through Generative Artificial Intelligence
- 25 April 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-23
Open AccessCARING-AI: Towards Authoring Context-aware Augmented Reality INstruction through Generative Artificial Intelligence
- Open Access
- 25 April 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-20
Transparent Barriers: Natural Language Access Control Policies for XR-Enhanced Everyday Objects
- 25 April 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-20
Open AccessTransparent Barriers: Natural Language Access Control Policies for XR-Enhanced Everyday Objects
- 24 February 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 31 (12) , 10200-10214
Visualizing Causality in Mixed Reality for Manual Task Learning: A Study
Mixed Reality (MR) is gaining prominence in manual task skill learning due to its in-situ, embodied, and immersive experience. To teach manual tasks, current methodologies break the task into hierarchies (tasks into subtasks) and visualize not only the current subtasks but also the future ones that are causally related. We investigate the impact of visualizing causality within an MR framework on manual task skill learning. We conducted a user study with 48 participants, experimenting with how presenting tasks in hierarchical causality levels (no causality, event-level, interaction-level, and gesture-level causality) affects user comprehension and performance in a complex assembly task. The research finds that displaying all causality levels enhances user understanding and task execution, with a compromise of learning time. Based on the results, we further provide design recommendations and in-depth discussions for future manual task learning systems.
- 24 February 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 31 (12) , 10200-10214
Visualizing Causality in Mixed Reality for Manual Task Learning: A Study
Mixed Reality (MR) is gaining prominence in manual task skill learning due to its in-situ, embodied, and immersive experience. To teach manual tasks, current methodologies break the task into hierarchies (tasks into subtasks) and visualize not only the current subtasks but also the future ones that are causally related. We investigate the impact of visualizing causality within an MR framework on manual task skill learning. We conducted a user study with 48 participants, experimenting with how presenting tasks in hierarchical causality levels (no causality, event-level, interaction-level, and gesture-level causality) affects user comprehension and performance in a complex assembly task. The research finds that displaying all causality levels enhances user understanding and task execution, with a compromise of learning time. Based on the results, we further provide design recommendations and in-depth discussions for future manual task learning systems.
- Open Access
- 11 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-16
avaTTAR: Table Tennis Stroke Training with Embodied and Detached Visualization in Augmented Reality
Table tennis stroke training is a critical aspect of player development. We designed a new augmented reality (AR) system, avaTTAR, for table tennis stroke training. The system provides both “on-body” (first-person view) and “detached” (third-person view) visual cues, enabling users to visualize target strokes and correct their attempts effectively with this dual perspectives setup. By employing a combination of pose estimation algorithms and IMU sensors, avaTTAR captures and reconstructs the 3D body pose and paddle orientation of users during practice, allowing real-time comparison with expert strokes. Through a user study, we affirm avaTTAR ’s capacity to amplify player experience and training results.
- 11 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-16
Open AccessavaTTAR: Table Tennis Stroke Training with Embodied and Detached Visualization in Augmented Reality
Table tennis stroke training is a critical aspect of player development. We designed a new augmented reality (AR) system, avaTTAR, for table tennis stroke training. The system provides both “on-body” (first-person view) and “detached” (third-person view) visual cues, enabling users to visualize target strokes and correct their attempts effectively with this dual perspectives setup. By employing a combination of pose estimation algorithms and IMU sensors, avaTTAR captures and reconstructs the 3D body pose and paddle orientation of users during practice, allowing real-time comparison with expert strokes. Through a user study, we affirm avaTTAR ’s capacity to amplify player experience and training results.
- 13 November 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 9 (1) , 451-458
Interacting Objects: A Dataset of Object-Object Interactions for Richer Dynamic Scene Representations
Dynamic environments in factories, surgical robotics, and warehouses increasingly involve humans, machines, robots, and various other objects such as tools, fixtures, conveyors, and assemblies. In these environments, numerous interactions occur not just between humans and objects but also between objects themselves. However, current scene-graph datasets predominantly focus on human-object interactions (HOI) and overlook object-object interactions (OOIs) despite the necessity of OOIs in effectively representing dynamic environments. This oversight creates a significant gap in the coverage of interactive elements in dynamic scenes. We address this gap by proposing, to the best of our knowledge, the first dataset* annotating for OOI categories in dynamic scenes. To model OOIs, we establish a classification taxonomy for spatio-temporal interactions. We use our taxonomy to annotate OOIs in video clips of dynamic scenes. Then, we introduce a spatio-temporal OOI classification task which aims to identify interaction categories between two given objects in a video clip. Further, we benchmark our dataset for the spatio-temporal OOI classification task by adopting state-of-the-art approaches from related areas of Human-Object Interaction Classification, Visual Relationship Classification, and Scene-Graph Generation. Additionally, we utilize our dataset to examine the effectiveness of OOI and HOI-based features in the context of Action Recognition. Notably, our experimental results show that OOI-based features outperform HOI-based features for the task of Action Recognition.
- 13 November 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 9 (1) , 451-458
Interacting Objects: A Dataset of Object-Object Interactions for Richer Dynamic Scene Representations
Dynamic environments in factories, surgical robotics, and warehouses increasingly involve humans, machines, robots, and various other objects such as tools, fixtures, conveyors, and assemblies. In these environments, numerous interactions occur not just between humans and objects but also between objects themselves. However, current scene-graph datasets predominantly focus on human-object interactions (HOI) and overlook object-object interactions (OOIs) despite the necessity of OOIs in effectively representing dynamic environments. This oversight creates a significant gap in the coverage of interactive elements in dynamic scenes. We address this gap by proposing, to the best of our knowledge, the first dataset* annotating for OOI categories in dynamic scenes. To model OOIs, we establish a classification taxonomy for spatio-temporal interactions. We use our taxonomy to annotate OOIs in video clips of dynamic scenes. Then, we introduce a spatio-temporal OOI classification task which aims to identify interaction categories between two given objects in a video clip. Further, we benchmark our dataset for the spatio-temporal OOI classification task by adopting state-of-the-art approaches from related areas of Human-Object Interaction Classification, Visual Relationship Classification, and Scene-Graph Generation. Additionally, we utilize our dataset to examine the effectiveness of OOI and HOI-based features in the context of Action Recognition. Notably, our experimental results show that OOI-based features outperform HOI-based features for the task of Action Recognition.