SonifyAR: Context-Aware Sound Effect Generation in Augmented Reality
Open Access
- 2 May 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-7
Abstract
Sound plays crucial roles in enhancing user experience and immersiveness in Augmented Reality (AR). However, current AR authoring platforms lack support for creating sound effects that harmonize with both the virtual and the real-world contexts. In this work, we present SonifyAR, a novel system for generating context-aware sound effects in AR experiences. SonifyAR implements a Programming by Demonstration (PbD) AR authoring pipeline. We utilize computer vision models and a large language model (LLM) to generate text descriptions that incorporate context information of user, virtual object and real world environment. This context information is then used to acquire sound effects with recommendation, generation, and retrieval methods. The acquired sound effects can be tested and assigned to AR events. Our user interface also provides the flexibility to allow users to iteratively explore and fine-tune the sound effects. We conducted a preliminary user study to demonstrate the effectiveness and usability of our system.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- A Dense Material Segmentation Dataset for Indoor and Outdoor Scene Parsing2022
- Towards Sound Accessibility in Virtual Reality2021
- The Effectiveness of Visual and Audio Wayfinding Guidance on Smartglasses for People with Low Vision2020
- Designing AR Visualizations to Facilitate Stair Navigation for People with Low Vision2019
- Sonic Interactions in Virtual Reality: State of the Art, Current Challenges, and Future DirectionsIEEE Computer Graphics and Applications, 2018
- Is Augmented Reality Technology an Effective Tool for E-commerce? An Interactivity and Vividness PerspectiveJournal of Interactive Marketing, 2017
- An experimental study of spatial sound usefulness in searching and navigating through AR environmentsVirtual Reality, 2015
- Auditory augmented reality: Object sonification for the visually impaired2012
- The Role of 3-D Sound in Human Reaction and Performance in Augmented Reality EnvironmentsIEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, 2007
- Programming by demonstration1998