CROSS-GAiT: Cross-Attention-Based Multimodal Representation Fusion for Parametric Gait Adaptation in Complex Terrains
Authors:
- 19 October 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 6079-6086
Abstract
We present CROSS-GAiT, a novel algorithm for quadruped robots that uses Cross Attention to fuse terrain representations derived from visual and time-series inputs; including linear accelerations, angular velocities, and joint efforts. These fused representations are used to continuously adjust two critical gait parameters (step height and hip splay), enabling adaptive gaits that respond dynamically to varying terrain conditions. To generate terrain representations, we process visual inputs through a masked Vision Transformer (ViT) encoder and time-series data through a dilated causal convolutional encoder. The Cross Attention mechanism then selects and integrates the most relevant features from each modality, combining terrain characteristics with robot dynamics for informed gait adaptation. This fused representation allows CROSS-GAiT to continuously adjust gait parameters in response to unpredictable terrain conditions in real-time. We train CROSS-GAiT on a diverse set of terrains including asphalt, concrete, brick pavements, grass, dense vegetation, pebbles, gravel, and sand and validate its generalization ability on unseen environments. Our hardware implementation on the Ghost Robotics Vision 60 demonstrates superior performance in challenging terrains, such as high-density vegetation, unstable surfaces, sandbanks, and deformable substrates. We observe at least a 7.04% reduction in IMU energy density and a 27.3% reduction in total joint effort, which directly correlates with increased stability and reduced energy usage when compared to state-of-the-art methods. Furthermore, CROSS-GAiT demonstrates at least a 64.5% increase in success rate and a 4.91% reduction in time to reach the goal in four complex scenarios. Additionally, the learned representations perform 4.48% better than the state-of-the-art on a terrain classification task.Keywords
All Related Versions
Funding Information
- Arm
This publication has 39 references indexed in Scilit:
- Multimodal Transformer Using Cross-Channel Attention For Object Detection In Remote Sensing Images2024
- Recursive Joint Cross-Modal Attention for Multimodal Fusion in Dimensional Emotion Recognition2024
- CrossFuse: A novel cross attention mechanism based infrared and visible image fusion approachInformation Fusion, 2023
- Coupling Vision and Proprioception for Navigation of Legged Robots2022
- Masked Autoencoders Are Scalable Vision Learners2022
- Is Cross-Attention Preferable to Self-Attention for Multi-Modal Emotion Recognition?2022
- Multi-Modality Cross Attention Network for Image and Sentence Matching2020
- Deep Texture Manifold for Ground Terrain Recognition2018
- Gaze and the Control of Foot Placement When Walking in Natural TerrainCurrent Biology, 2018
- Effects of turning gait parameters on energy consumption and stability of a six-legged walking robotRobotics and Autonomous Systems, 2012