Zackary P. T. Sin
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Research Interests
Zackary P. T. Sin has a total of 40 co-authors. Here are the top co-authors by number of shared publications:
- Peter Hiu Fung Ng
Department of Rehabilitation Sciences, The Hong Kong Polytechnic University, Hong Kong SAR, China
9 shared publications - Chen Li
The Hong Kong Polytechnic University
7 shared publications - Qing Li
The Hong Kong Polytechnic University, Hong Kong SAR
7 shared publications - Xiao Huang
The Hong Kong Polytechnic University, Hong Kong, Hong Kong
7 shared publications - George Baciu
Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China
6 shared publications - Ye Jia
The Hong Kong Polytechnic University, Hong Kong SAR, China
6 shared publications
Latest Publications
- Open Access
- 2 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-13
Seeing Beyond the Chart: Behaviour-Aware Self-Directed Visual Acuity Screening with Multimodal Interaction
- 2 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-13
Open AccessSeeing Beyond the Chart: Behaviour-Aware Self-Directed Visual Acuity Screening with Multimodal Interaction
- Open Access
- 15 July 2026
- Published by Elsevier in SSRN Electronic Journal
Every Second Matters: Investigating the Effects of Conversational Response Delays and Contextualized Fillers for Digital Avatars
Emergent large language model (LLM)-based digital avatars are poised to have broad applications across domains. However, LLMs, together with auxiliary services such as text-to-speech, require processing time and can generate undesirable conversational response delays. There is still limited understanding of how such delays affect human-avatar interaction, particularly the point at which a delay changes from acceptable to unacceptable. In this paper, we first conduct a preliminary study in virtual reality (N = 30) to investigate the effects of response delays ranging from 0 to 16000 ms. The results show that delays between 0 and 2000 ms are generally acceptable, with around 1000 ms perceived as the most natural. Noticeable degradation begins by 4000 ms and becomes pronounced at 8000 ms. Based on these findings, we implemented a turn-taking scheme that uses a lightweight LLM to generate contextualized conversational fillers within 1000 ms. We then compare this approach with commonly used canned lexical and non-lexical fillers in a user study (N = 63). Canned non-lexical fillers provide no improvement and can worsen the experience, while canned lexical fillers offer only limited benefits. In contrast, contextualized fillers more effectively preserve conversational realism, social context, interactivity, anthropomorphism, and animacy, while reducing discomfort. These findings show that faster responses are not always better and that filler effectiveness depends on contextual relevance. This work provides design guidance for more natural and socially appropriate digital avatars and conversational user interfaces.
- 15 July 2026
- Published by Elsevier in SSRN Electronic Journal
Open AccessEvery Second Matters: Investigating the Effects of Conversational Response Delays and Contextualized Fillers for Digital Avatars
Emergent large language model (LLM)-based digital avatars are poised to have broad applications across domains. However, LLMs, together with auxiliary services such as text-to-speech, require processing time and can generate undesirable conversational response delays. There is still limited understanding of how such delays affect human-avatar interaction, particularly the point at which a delay changes from acceptable to unacceptable. In this paper, we first conduct a preliminary study in virtual reality (N = 30) to investigate the effects of response delays ranging from 0 to 16000 ms. The results show that delays between 0 and 2000 ms are generally acceptable, with around 1000 ms perceived as the most natural. Noticeable degradation begins by 4000 ms and becomes pronounced at 8000 ms. Based on these findings, we implemented a turn-taking scheme that uses a lightweight LLM to generate contextualized conversational fillers within 1000 ms. We then compare this approach with commonly used canned lexical and non-lexical fillers in a user study (N = 63). Canned non-lexical fillers provide no improvement and can worsen the experience, while canned lexical fillers offer only limited benefits. In contrast, contextualized fillers more effectively preserve conversational realism, social context, interactivity, anthropomorphism, and animacy, while reducing discomfort. These findings show that faster responses are not always better and that filler effectiveness depends on contextual relevance. This work provides design guidance for more natural and socially appropriate digital avatars and conversational user interfaces.
- 2 July 2026
- Published by Springer Nature
- p. 41-55
PockeTA: Virtual Teaching Assistant Platform with Course Context
- 2 July 2026
- Published by Springer Nature
- p. 41-55
PockeTA: Virtual Teaching Assistant Platform with Course Context
- Open Access
- 14 December 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-3
Sketch-in-Scene: Coherent 3D Scene Asset Generation with Sketches
- 14 December 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-3
Open AccessSketch-in-Scene: Coherent 3D Scene Asset Generation with Sketches
- Open Access
- 27 September 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-18
Can You Move These Over There? Exploring an LLM-based VR Mover to Support Natural Multi-object Manipulation
- 27 September 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-18
Open AccessCan You Move These Over There? Exploring an LLM-based VR Mover to Support Natural Multi-object Manipulation
- 26 August 2025
- Published by Wiley in Computer Graphics Forum
- Vol. 44 (6)
3DGM: Deformable and Texturable 3D Gaussian Model via Level‐of‐Detail Proxy
3D Gaussian Splatting has markedly impacted neural rendering by achieving impressive fidelity and performance. Despite this achievement, it is not readily applicable to developing interactive applications. Real‐time applications like XR apps and games require functions such as animation, UV mapping and level of detail (LOD) simultaneously manipulated through a 3D model. To address this need, we propose a modelling strategy analogous to typical 3D models, which we call 3D Gaussian Model (3DGM). 3DGM relies on attaching 3D Gaussians on the triangles of a mesh proxy, and the key idea is to bind sheared 3D Gaussians in texture space and re‐projecting them back to world space through implicit shell mapping; this design naturally enables deformation and UV mapping via the proxy. Further, to optimize speed and fidelity based on different viewing distances, each triangle can be tessellated to change the number of involved 3D Gaussians adaptively. Application‐wise, we will show that our proxy‐based 3DGM is capable of enabling novel deformation without animated training data, texture transferring via UV mapping of the 3D Gaussians, and LOD rendering. The results indicate that our model achieves better fidelity for deformation and better optimization of fidelity and performance given different viewing distances. Further, we believe the results indicate the potential of our work for enabling interactive applications for 3D Gaussian Splatting.
- 26 August 2025
- Published by Wiley in Computer Graphics Forum
- Vol. 44 (6)
3DGM: Deformable and Texturable 3D Gaussian Model via Level‐of‐Detail Proxy
3D Gaussian Splatting has markedly impacted neural rendering by achieving impressive fidelity and performance. Despite this achievement, it is not readily applicable to developing interactive applications. Real‐time applications like XR apps and games require functions such as animation, UV mapping and level of detail (LOD) simultaneously manipulated through a 3D model. To address this need, we propose a modelling strategy analogous to typical 3D models, which we call 3D Gaussian Model (3DGM). 3DGM relies on attaching 3D Gaussians on the triangles of a mesh proxy, and the key idea is to bind sheared 3D Gaussians in texture space and re‐projecting them back to world space through implicit shell mapping; this design naturally enables deformation and UV mapping via the proxy. Further, to optimize speed and fidelity based on different viewing distances, each triangle can be tessellated to change the number of involved 3D Gaussians adaptively. Application‐wise, we will show that our proxy‐based 3DGM is capable of enabling novel deformation without animated training data, texture transferring via UV mapping of the 3D Gaussians, and LOD rendering. The results indicate that our model achieves better fidelity for deformation and better optimization of fidelity and performance given different viewing distances. Further, we believe the results indicate the potential of our work for enabling interactive applications for 3D Gaussian Splatting.
- 2 November 2024
- Published by Elsevier in International Journal of Human-Computer Studies
- Vol. 194
Traceable teleportation: Improving spatial learning in virtual locomotion
- 2 November 2024
- Published by Elsevier in International Journal of Human-Computer Studies
- Vol. 194
Traceable teleportation: Improving spatial learning in virtual locomotion
- Open Access
- 28 May 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Learning Technologies
- Vol. 17, 1794-1808
Knowledge-Graph-Driven Mind Mapping for Immersive Collaborative Learning: A Pilot Study in Edu-Metaverse
One of the promises of edu-metaverse is its ability to provide a virtual environment that enables us to engage in learning activities that are similar to or on par with reality. The digital enhancements introduced in a virtual environment contribute to our increased expectations of novel learning experiences. However, despite its promising outcomes, there appears to be limited adoption of the edu-metaverse for practical learning at this time. We believe this can be attributed to the fact that there is a lack of investigation into learners' behavior given a social learning environment. This lack of investigation is critical, as without behavioral insight, it hinders the development of education material and the direction of an edu-metaverse. This paper aims to provide insights into learner behavior in edu-metaverse by conducting a focused pilot study. Particularly, we investigate the learners' behavior when drawing mind maps in the metaverse aided by the embedded knowledge graph. Compared to Zoom, a typical video conferencing and remote collaboration platform, it seems that users are more involved in learning activities, and the co-presence seems to assist in engagement so that they are more willing to contribute to the learning task collaboratively. Our work also provides insight into the interaction and learning activity design for edu-metaverse, particularly when mind maps are involved.
- 28 May 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Learning Technologies
- Vol. 17, 1794-1808
Open AccessKnowledge-Graph-Driven Mind Mapping for Immersive Collaborative Learning: A Pilot Study in Edu-Metaverse
One of the promises of edu-metaverse is its ability to provide a virtual environment that enables us to engage in learning activities that are similar to or on par with reality. The digital enhancements introduced in a virtual environment contribute to our increased expectations of novel learning experiences. However, despite its promising outcomes, there appears to be limited adoption of the edu-metaverse for practical learning at this time. We believe this can be attributed to the fact that there is a lack of investigation into learners' behavior given a social learning environment. This lack of investigation is critical, as without behavioral insight, it hinders the development of education material and the direction of an edu-metaverse. This paper aims to provide insights into learner behavior in edu-metaverse by conducting a focused pilot study. Particularly, we investigate the learners' behavior when drawing mind maps in the metaverse aided by the embedded knowledge graph. Compared to Zoom, a typical video conferencing and remote collaboration platform, it seems that users are more involved in learning activities, and the co-presence seems to assist in engagement so that they are more willing to contribute to the learning task collaboratively. Our work also provides insight into the interaction and learning activity design for edu-metaverse, particularly when mind maps are involved.
- Open Access
- 29 June 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Learning Technologies
- Vol. 16 (6) , 1096-1110
Toward an Edu-Metaverse of Knowledge: Immersive Exploration of University Courses
Metaverse, an alternative universe for play, work and interaction, has become a captivating topic for academia and industry in recent times. This opens the question on what a metaverse for education, or edu-metaverse, should look like. It is believed that this metaverse for learning should be grounded by a pedagogical theory. Particularly, we propose a constructivist metaverse learning theory with eight actionable principles to guide the edu-metaverse and its applications. With this metaverse learning theory, we further propose the framework for an edu-metaverse; it is essentially walkable yellow pages that connect knowledge. The core idea is to combine the structure of knowledge graphs and the immersion of virtual reality in order to facilitate association, exploration and engagement in learning. Our current prototype for this edu-metaverse vision, K-Cube VR, is also presented. We have tested K-Cube VR for the introduction of course topics to our students and the results indicate that our edu-metaverse framework benefits students by providing a focused environment and structured learning on the topics of a course, akin to a mind map. Overall, in this paper, we present an edu-metaverse design that is rooted in a constructivist pedagogy that already shows promising results from a pilot user study via our metaverse prototype.
- 29 June 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Learning Technologies
- Vol. 16 (6) , 1096-1110
Open AccessToward an Edu-Metaverse of Knowledge: Immersive Exploration of University Courses
Metaverse, an alternative universe for play, work and interaction, has become a captivating topic for academia and industry in recent times. This opens the question on what a metaverse for education, or edu-metaverse, should look like. It is believed that this metaverse for learning should be grounded by a pedagogical theory. Particularly, we propose a constructivist metaverse learning theory with eight actionable principles to guide the edu-metaverse and its applications. With this metaverse learning theory, we further propose the framework for an edu-metaverse; it is essentially walkable yellow pages that connect knowledge. The core idea is to combine the structure of knowledge graphs and the immersion of virtual reality in order to facilitate association, exploration and engagement in learning. Our current prototype for this edu-metaverse vision, K-Cube VR, is also presented. We have tested K-Cube VR for the introduction of course topics to our students and the results indicate that our edu-metaverse framework benefits students by providing a focused environment and structured learning on the topics of a course, akin to a mind map. Overall, in this paper, we present an edu-metaverse design that is rooted in a constructivist pedagogy that already shows promising results from a pilot user study via our metaverse prototype.