CROSS-GAiT: Cross-Attention-Based Multimodal Representation Fusion for Parametric Gait Adaptation in Complex Terrains
Authors:
- Gershom SeneviratneGSGershom Seneviratne
- Kasun WeerakoonKWKasun Weerakoon
- Mohamed ElnoorMEMohamed Elnoor
- Vignesh RajgopalVRVignesh Rajgopal
- Harshavarthan VaratharajanHVHarshavarthan Varatharajan
- Mohamed Khalid M JaffarMJMohamed Khalid M Jaffar
- Jason PuseyJPJason Pusey
- Dinesh ManochaDMDinesh Manocha
Preprint
Abstract
We present CROSS-GAiT, a novel algorithm for quadruped robots that uses Cross Attention to fuse terrain representations derived from visual and time-series inputs, including linear accelerations, angular velocities, and joint efforts. These fused representations are used to adjust the robot's step height and hip splay, enabling adaptive gaits that respond dynamically to varying terrain conditions. We generate these terrain representations by processing visual inputs through a masked Vision Transformer (ViT) encoder and time-series data through a dilated causal convolutional encoder. The cross-attention mechanism then selects and integrates the most relevant features from each modality, combining terrain characteristics with robot dynamics for better-informed gait adjustments. CROSS-GAiT uses the combined representation to dynamically adjust gait parameters in response to varying and unpredictable terrains. We train CROSS-GAiT on data from diverse terrains, including asphalt, concrete, brick pavements, grass, dense vegetation, pebbles, gravel, and sand. Our algorithm generalizes well and adapts to unseen environmental conditions, enhancing real-time navigation performance. CROSS-GAiT was implemented on a Ghost Robotics Vision 60 robot and extensively tested in complex terrains with high vegetation density, uneven/unstable surfaces, sand banks, deformable substrates, etc. We observe at least a 7.04% reduction in IMU energy density and a 27.3% reduction in total joint effort, which directly correlates with increased stability and reduced energy usage when compared to state-of-the-art methods. Furthermore, CROSS-GAiT demonstrates at least a 64.5% increase in success rate and a 4.91% reduction in time to reach the goal in four complex scenarios. Additionally, the learned representations perform 4.48% better than the state-of-the-art on a terrain classification task.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- Multimodal Transformer Using Cross-Channel Attention For Object Detection In Remote Sensing Images2024
- AMCO: Adaptive Multimodal Coupling of Vision and Proprioception for Quadruped Robot Navigation in Outdoor Environments2024
- BronchoCopilot: Towards Autonomous Robotic Bronchoscopy via Multimodal Reinforcement Learning2024
- Recursive Joint Cross-Modal Attention for Multimodal Fusion in Dimensional Emotion Recognition2024
- CrossFuse: A novel cross attention mechanism based infrared and visible image fusion approachInformation Fusion, 2023
- GA-Nav: Efficient Terrain Segmentation for Robot Navigation in Unstructured Outdoor EnvironmentsIEEE Robotics and Automation Letters, 2022
- BADGR: An Autonomous Self-Supervised Learning-Based Navigation SystemIEEE Robotics and Automation Letters, 2021
- Where Should I Walk? Predicting Terrain Properties From Images Via Self-Supervised LearningIEEE Robotics and Automation Letters, 2019
- Gaze and the Control of Foot Placement When Walking in Natural TerrainCurrent Biology, 2018
- Effects of turning gait parameters on energy consumption and stability of a six-legged walking robotRobotics and Autonomous Systems, 2012