Yuya Okadome

As expectations for computer graphic (CG) avatars and conversational robots increase, enhancing dialogue skills via multimodal channels is crucial for achieving fluent interactions with humans. Thus, automatic interaction motion generation is essential for autonomous conversation systems. Natural motion generation, such as appropriate nodding, requires considering the behavior and voice of the conversation partner. However, current models generate motion from audio or text, neglecting interaction factors. In this study, we implemented an interaction diffusion model (IDM) that uses a diffusion approach and masking features to generate interaction behaviors for dyadic conversation. IDM accounts for two participants, using masks to generate features from conditional inputs. This allows for accommodating conditions like missing features and forecasting without retraining. The experimental results suggests that the model generates the human-like behaviors during conversation in 30 ms.

Mobility in daily unstructured environments, particularly in human-centered scenarios, remains a fundamental challenge for mobile robots. While traditional prediction-based approaches primarily estimate partial features for robot decision making, such as position and velocity, recent world models enable direct prediction of future sensory data. However, their potentials in human-inclusive environments remain underexplored. To assess the feasibility of world models in facilitating human-robot interactions, we propose a robot framework using a deep generative model that jointly predicts multiple future observations and actions. Our approach leverages first-person-view (FPV) raw sensor data, integrating both observations and actions to enhance predictive capabilities in dynamic human-populated settings. Experimental results demonstrate that our method is capable of generating a range of candidate futures for one condition and planning actions based on observation guidance. These findings highlight the potential of our approach for facilitating autonomous robots’ coexistence with human.

Acoustic data inherently contains spatial information regarding sounds. Most robotic systems rely primarily on speech recognition results for control; however, the utilization of rich directional cues could enhance their functionality. Especially in human-robot interaction, both verbal content and spatial context in acoustic data are crucial. To capture such auditory context, we employ acoustic maps in this research. These maps represent soundscapes as 2D projections, allowing robots to integrate auditory spatial cues with visual perception. Using a custom acoustic-visual module, we collect egocentric data from four daily scenarios and generate corresponding acoustic maps. By demonstrating a predictive sound source positioning task, we confirm the potential of acoustic maps to capture dialogue-relevant information. Based on these findings, we anticipate potential applications for robots, such as identifying individuals attempting to engage with the robot in crowded environments or predicting turn-taking in multi-party conversations.

Recent advancements in deep learning and large-scale language models have driven interest in creating conversational robots capable of heartful interactions with humans. However, most existing systems are tailored for specific contexts and lack a comprehensive framework to assist humans in daily activities. Developing a fully autonomous robot that can operate alongside humans and interact with them in dynamic environments is a complex task that requires addressing particular software and hardware demands. These demands often conflict, resulting in trade-offs based on the intended application. To overcome this challenge, we propose a body design that leverages passive dynamics. We illustrate how the essential devices for computation, sensing, and actuation can be integrated into the robot without compromising its embodiment requirements. Conversely, we utilize the weight and placement of these devices to achieve the physical characteristics needed for a robotic body that is well-suited for coexistence with humans. We validate this approach through the development of Indy, a mobile communication robot platform designed to coexist with humans. The employed mechanisms enable Indy to maintain an upright posture without active control and ensure safe physical interactions. Experimental results indicate that the generated body movement positively impacts users’ perceptions, promoting a sense of anthropomorphism and enhancing the interaction experience. This confirms that the selected design approach not only provides mechanical advantages but also improves the natural appearance of the robot’s movements.

Systems in the real world often consist of multiple subsystems interacting with each other, for example, the musculoskeletal system or human-human interaction. The measurement of temporal changes in these systems involves a multidimensional time series. This study introduces a novel framework for extracting features from multidimensional time series data with a group structure using self-supervised learning techniques. Specifically, we use a “lag operation,” which is a temporal shifting operation applied to the features of a certain group. We propose a self-supervised learning method for a neural network model that uses the data automatically generated by the lag operation and its corresponding operation labels to capture and quantify the interaction between groups. Upon completion of the training process, the representation space is obtained with the expectation that it will capture timing-dependent features within its boundaries. We define and calculate the interaction score, R-score, on the obtained space. To validate our approach, we apply the proposed methodology to an artificial oscillator and approximately 4 hours of conversational data to evaluate the R-score properties. From the results of the artificial data, the R-score increases when the connection between the groups is large. From the high R-score region of the representation space of the conversation data, we extract the data that contain social behaviors such as “eye contact,” “turn-taking,” and “smiling,” which are related to the interaction between the participants. The experimental results suggest that the proposed method can obtain a representation space for time series data with a group structure.