A Feasibility Study With In-the-Wild Data in Human Interaction Settings: Acoustic-Visual Fusion for Predictive Sound Source Positioning
Open Access
- 26 August 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 13, 150365-150378
Abstract
Acoustic data inherently contains spatial information regarding sounds. Most robotic systems rely primarily on speech recognition results for control; however, the utilization of rich directional cues could enhance their functionality. Especially in human-robot interaction, both verbal content and spatial context in acoustic data are crucial. To capture such auditory context, we employ acoustic maps in this research. These maps represent soundscapes as 2D projections, allowing robots to integrate auditory spatial cues with visual perception. Using a custom acoustic-visual module, we collect egocentric data from four daily scenarios and generate corresponding acoustic maps. By demonstrating a predictive sound source positioning task, we confirm the potential of acoustic maps to capture dialogue-relevant information. Based on these findings, we anticipate potential applications for robots, such as identifying individuals attempting to engage with the robot in crowded environments or predicting turn-taking in multi-party conversations.Keywords
Funding Information
- Japan Science and Technology Agency (JST) Moonshot Research and Development (JPMJMS2011)
This publication has 28 references indexed in Scilit:
- DOA-Aware Audio-Visual Self-Supervised Learning for Sound Event Localization and Detection2023
- STAR Loss: Reducing Semantic Ambiguity in Facial Landmark Detection2023
- Multi-speaker DoA Estimation Using Audio and Visual ModalityNeural Processing Letters, 2023
- Estimation of continuous valence and arousal levels from faces in naturalistic conditionsNature Machine Intelligence, 2021
- A Python framework for microphone array data processingApplied Acoustics, 2017
- Deep clustering: Discriminative embeddings for segmentation and separation2016
- Visualizing and Understanding Convolutional Networks2014
- Active-speaker detection and localization with microphones and cameras embedded into a robotic head2013
- Algorithms for acoustic localization based on microphone array in service roboticsRobotics and Autonomous Systems, 2003
- Beamforming: a versatile approach to spatial filteringIEEE ASSP Magazine, 1988