UbiHearo: Bringing Scenario-Aware Sound Guidance to DHH Users with Mobile Agents
Open Access
- 30 September 2026
- journal article
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
- Vol. 10 (3) , 1-27
Abstract
For Deaf and Hard-of-Hearing (DHH) users, effective sound awareness goes beyond detecting what sound occurs; it requires understanding what it means, how urgent it is , and what to do. However, existing mobile assistants largely reduce acoustic scenes to isolated labels or generic alerts, failing to provide contextual guidance when the same sound implies different risks across scenarios. We present UbiHearo, a mobile hearing assistant agent that closes the sensing-reasoning-action loop for scenario-aware and personalized DHH assistance. UbiHearo adopts a neuro-symbolic design that transforms continuous acoustic and physical context streams into structured intermediate representations (IRs) for controllable reasoning and guidance. It introduces three key mechanisms. First , a perception-thresholded sensing mechanism enables energy-efficient always-on awareness by activating processing only for perceptually meaningful events while preserving safety-critical cues through deterministic overrides. Second , a staged neuro-symbolic reasoning framework performs promptless scenario understanding by conditioning an on-device audio-language model on structured IRs instead of user-authored context descriptions. Third , a bounded human-in-the-loop personalization mechanism adapts user-specific guidance through lightweight on-device decision-boundary updates while preserving population-aligned reasoning. We evaluate UbiHearo on smartphones and in-vehicle systems across diverse scenarios. It achieves favorable accuracy-latency-energy tradeoffs and personalized guidance.Keywords
Funding Information
- National Natural Science Foundation of China (62522215, 62532009, 62472354, U25B2040, 62272010)
- CityU APRC grant (No. 9610633)
This publication has 15 references indexed in Scilit:
- Penetrative AI: Making LLMs Comprehend the Physical World2024
- AdaptiveSound: An Interactive Feedback-Loop System to Improve Sound Recognition for Deaf and Hard of Hearing Users2023
- Haptic-Captioning: Using Audio-Haptic Interfaces to Enhance Speaker Indication in Real-Time Captions for Deaf and Hard-of-Hearing Viewers2023
- Making sense of sensory language: Acquisition of sensory knowledge by individuals with congenital sensory impairmentsNeuropsychologia, 2022
- SoundWatch: Exploring Smartwatch-based Deep Learning Approaches to Support Sound Awareness for Deaf and Hard of Hearing Users2020
- Augmenting Conversational Agents with Ambient Acoustic Contexts2020
- Exploring Sound Awareness in the Home for People who are Deaf or Hard of Hearing2019
- Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference2018
- Audio Set: An ontology and human-labeled dataset for audio events2017
- Environmental sound classification with convolutional neural networks2015