Long-horizon Locomotion and Manipulation on a Quadrupedal Robot with Large Language Models
Authors:
- Yutao OuyangYOYutao Ouyang
- Jinhan LiJLJinhan Li
- Yunfei LiYLYunfei Li
- Zhongyu LiZLZhongyu Li
- Chao YuCYChao Yu
- Koushil SreenathKSKoushil Sreenath
- Yi WuYWYi Wu
Preprint
Abstract
We present a large language model (LLM) based system to empower quadrupedal robots with problem-solving abilities for long-horizon tasks beyond short-term motions. Long-horizon tasks for quadrupeds are challenging since they require both a high-level understanding of the semantics of the problem for task planning and a broad range of locomotion and manipulation skills to interact with the environment. Our system builds a high-level reasoning layer with large language models, which generates hybrid discrete-continuous plans as robot code from task descriptions. It comprises multiple LLM agents: a semantic planner for sketching a plan, a parameter calculator for predicting arguments in the plan, and a code generator to convert the plan into executable robot code. At the low level, we adopt reinforcement learning to train a set of motion planning and control skills to unleash the flexibility of quadrupeds for rich environment interactions. Our system is tested on long-horizon tasks that are infeasible to complete with one single skill. Simulation and real-world experiments show that it successfully figures out multi-step strategies and demonstrates non-trivial behaviors, including building tools or notifying a human for help.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Learning Agile Bipedal Motions on a Quadrupedal Robot2024
- Extreme Parkour with Legged Robots2024
- Pedipulate: Enabling Manipulation Skills using a Quadruped Robot’s Leg2024
- Learning Whole-Body Manipulation for Quadrupedal RobotIEEE Robotics and Automation Letters, 2023
- Legs as Manipulator: Pushing Quadrupedal Agility Beyond Locomotion2023
- Long-Horizon Multi-Robot Rearrangement Planning for Construction AssemblyIEEE Transactions on Robotics, 2022
- Learning robust perceptive locomotion for quadrupedal robots in the wildScience Robotics, 2022
- Integrated Task and Motion PlanningAnnual Review of Control, Robotics, and Autonomous Systems, 2021
- Learning quadrupedal locomotion over challenging terrainScience Robotics, 2020
- Learning agile and dynamic motor skills for legged robotsScience Robotics, 2019