Shantanu Vyas

Immersive extended reality (XR) technologies, including augmented reality (AR), virtual reality, and mixed reality, are transforming the landscape of education and training through experiences that promote skill acquisition and enhance memory retention. These technologies have notably improved decision making and situational awareness in public safety training. Despite the promise of these advancements, XR adoption for emergency response has been slow. This hesitancy can be partially attributed to a lack of guidance for integrating these novel technologies into existing curricula. This work aims to guide instructional designers, curriculum developers, and technologists in seamlessly integrating immersive technologies into public safety training curricula. This work provides a comprehensive account of our collaboration with instructional designers, public safety personnel, and subject matter experts in developing an AR-based training curriculum for the Sort, Assess, Life-saving Interventions, Treatment/Transport triage technique used in mass casualty incidents (MCIs). In addition, we introduce a systematic framework for public safety curriculum development based on the Analyze, Design, Develop, Implement, Evaluate instructional design model. Leveraging a human-centered design approach, we first analyze the necessity for immersive learning in public safety. Next, we identify the obstacles in developing XR training experiences and outline our construct of a training prototype through iterative evaluations based on stakeholder feedback. Finally, we share qualitative insights through iterative evaluations with firefighters and emergency medical technicians performing MCI triage tasks in AR, supplemented by survey questionnaires and semistructured interviews. Our goal is to provide a blueprint for a successful integration of immersive technologies into public safety training curricula.

We investigate speech-based input as a means to enable reflective thinking for younger individuals (middle- and high-school students) during design iterations. Verbalization offers a unique way to externalize ideas in early design and could therefore lead to new pathways for exploration and iteration, especially for K-12 students who possess the creative potential but are not technically trained in the design process. Interactive design systems, however, by-and-large utilize sketching, multitouch, and gestural inputs. As a result, (1) there is little know-how regarding how to operationalize verbal inputs as a meaningful way to facilitate idea exploration and (2) there is little fundamental understanding of the underlying cognitive mechanisms for iteration through verbal communication. We take the initial steps toward these gaps by first designing and implementing the ShapOrator interface that takes verbal descriptions of geometric parameters (shape, size, instances) in a semi-natural language form and determines the appropriate transformations to a given design artifact modeled as a shape assembly. By using ShapOrator as our experimental setup, we conducted an in-depth observational study on ten middle- and high-school students tasked with designing spaceships. Our study revealed that participants were able to create a variety of designs while associating functional and topical contexts to their spaceships throughout the design iteration process.

We explore verbalization as a means for quick-and-dirty 3D shape exploration in early design. Our work stems from the knowledge gap that the fundamental principles necessary to operationalize speech as a viable means for describing and communicating 3D digital forms do not currently exist. To address this gap, we present a case study on 3D scene modeling within the context of interior design. For this, we implemented a constrained workflow wherein a user can iteratively generate variations of seed templates of objects in a scene through verbal input. By using this workflow as an experimental setup, we systematically study four aspects of speech-based shape exploration, namely, (a) design-in-context (creating one shape with respect or in relation to the other), (b) order independence (sequence of parts preferred in speech-based shape exploration), (c) multiscale exploration (study how speech allows overview-then-detail modifications), and (d) semantic regions of interest (effectiveness of speech for modifying regions of a given object). We finally present an observational study with six participants selected from diverse backgrounds to better understand shape verbalization.