Robot Sound Interpretation: Combining Sight and Sound in Learning-Based Control
Authors:
- 24 October 2020
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 5580-5587
Abstract
We explore the interpretation of sound for robot decision making, inspired by human speech comprehension. While previous methods separate sound processing unit and robot controller, we propose an end-to-end deep neural network which directly interprets sound commands for visual-based decision making. The network is trained using reinforcement learning with auxiliary losses on the sight and sound networks. We demonstrate our approach on two robots, a TurtleBot3 and a Kuka-IIWA arm, which hear a command word, identify the associated target object, and perform precise control to reach the target. For both robots, we show the effectiveness of our network in generalization to sound types and robotic tasks empirically. We successfully transfer the policy learned in simulator to a real-world TurtleBot3.Keywords
All Related Versions
This publication has 29 references indexed in Scilit:
- Effective Approaches to Attention-based Neural Machine Translation2015
- A Dataset and Taxonomy for Urban Sound Research2014
- Developing embodied cognition: insights from children’s concepts and language processingFrontiers in Psychology, 2014
- Two-handed gesture recognition and fusion with speech to command a robotAutonomous Robots, 2011
- Neural Processing of Auditory Signals and Modular Neural Control for Sound Tropism of Walking MachinesInternational Journal of Advanced Robotic Systems, 2005
- The Development of Embodied Cognition: Six Lessons from BabiesArtificial Life, 2005
- Grounded auditory development by a developmental robot2002
- Robot speech learning via entropy guided LVQ and memory association2002
- Towards a Developmental Cognitive ScienceAnnals of the New York Academy of Sciences, 1990
- Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentencesIEEE Transactions on Acoustics, Speech, and Signal Processing, 1980