RoboGround: Robotic Manipulation with Grounded Vision-Language Priors
Authors:
- Haifeng HuangHHHaifeng Huang
- Yilun ChenYCYilun Chen
- Hao LiHLHao Li
Preprint
Abstract
Recent advancements in robotic manipulation have highlighted the potential of intermediate representations for improving policy generalization. In this work, we explore grounding masks as an effective intermediate representation, balancing two key advantages: (1) effective spatial guidance that specifies target objects and placement areas while also conveying information about object shape and size, and (2) broad generalization potential driven by large-scale vision-language models pretrained on diverse grounding datasets. We introduce RoboGround, a grounding-aware robotic manipulation system that leverages grounding masks as an intermediate representation to guide policy networks in object manipulation tasks. To further explore and enhance generalization, we propose an automated pipeline for generating large-scale, simulated data with a diverse set of objects and instructions. Extensive experiments show the value of our dataset and the effectiveness of grounding masks as intermediate guidance, significantly enhancing the generalization abilities of robot policies.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- The All-Seeing Project V2: Towards General Relation Comprehension of the Open World2024
- RoboCasa: Large-Scale Simulation of Household Tasks for Generalist Robots2024
- Any-point Trajectory Modeling for Policy Learning2024
- MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting2024
- Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware2023
- Diffusion Policy: Visuomotor Policy Learning via Action Diffusion2023
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning2023
- FlowBot3D: Learning 3D Articulation Flow to Manipulate Articulated Objects2022
- CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation TasksIEEE Robotics and Automation Letters, 2022
- RLBench: The Robot Learning Benchmark & Learning EnvironmentIEEE Robotics and Automation Letters, 2020