SonifyAR: Context-Aware Sound Generation in Augmented Reality
Open Access
- 11 October 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-13
Abstract
Sound plays a crucial role in enhancing user experience and immersiveness in Augmented Reality (AR). However, current platforms lack support for AR sound authoring due to limited interaction types, challenges in collecting and specifying context information, and difficulty in acquiring matching sound assets. We present SonifyAR, an LLM-based AR sound authoring system that generates context-aware sound effects for AR experiences. SonifyAR expands the current design space of AR sound and implements a Programming by Demonstration (PbD) pipeline to automatically collect contextual information of AR events, including virtual-content-semantics and real-world context. This context information is then processed by a large language model to acquire sound effects with Recommendation, Retrieval, Generation, and Transfer methods. To evaluate the usability and performance of our system, we conducted a user study with eight participants and created five example applications, including an AR-based science experiment, and an assistive application for low-vision AR users.Keywords
All Related Versions
Funding Information
- NSF (National Science Foundation) (1763199)
This publication has 37 references indexed in Scilit:
- Taxonomy and definition of audio augmented reality (AAR): A grounded theory studyInternational Journal of Human-Computer Studies, 2023
- Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation2023
- I Hear Your True Colors: Image Guided Audio Generation2023
- A Dense Material Segmentation Dataset for Indoor and Outdoor Scene Parsing2022
- Augmented reality in STEM education: a systematic reviewInteractive Learning Environments, 2020
- The Trouble with Augmented Reality/Virtual Reality Authoring Tools2018
- Visual to Sound: Generating Natural Sound for Videos in the Wild2018
- Augmented Reality (AR) Technology on The Android Operating System in Chemistry LearningIOP Conference Series: Materials Science and Engineering, 2018
- AR4VI: AR as an Accessibility Tool for People with Visual Impairments2017
- An experimental study of spatial sound usefulness in searching and navigating through AR environmentsVirtual Reality, 2015