Chenfei Xu

Mobility in daily unstructured environments, particularly in human-centered scenarios, remains a fundamental challenge for mobile robots. While traditional prediction-based approaches primarily estimate partial features for robot decision making, such as position and velocity, recent world models enable direct prediction of future sensory data. However, their potentials in human-inclusive environments remain underexplored. To assess the feasibility of world models in facilitating human-robot interactions, we propose a robot framework using a deep generative model that jointly predicts multiple future observations and actions. Our approach leverages first-person-view (FPV) raw sensor data, integrating both observations and actions to enhance predictive capabilities in dynamic human-populated settings. Experimental results demonstrate that our method is capable of generating a range of candidate futures for one condition and planning actions based on observation guidance. These findings highlight the potential of our approach for facilitating autonomous robots’ coexistence with human.

Acoustic data inherently contains spatial information regarding sounds. Most robotic systems rely primarily on speech recognition results for control; however, the utilization of rich directional cues could enhance their functionality. Especially in human-robot interaction, both verbal content and spatial context in acoustic data are crucial. To capture such auditory context, we employ acoustic maps in this research. These maps represent soundscapes as 2D projections, allowing robots to integrate auditory spatial cues with visual perception. Using a custom acoustic-visual module, we collect egocentric data from four daily scenarios and generate corresponding acoustic maps. By demonstrating a predictive sound source positioning task, we confirm the potential of acoustic maps to capture dialogue-relevant information. Based on these findings, we anticipate potential applications for robots, such as identifying individuals attempting to engage with the robot in crowded environments or predicting turn-taking in multi-party conversations.