KALIE: Fine-Tuning Vision-Language Models for Open-World Manipulation without Robot Data
Authors:
Preprint
Abstract
Building generalist robotic systems involves effectively endowing robots with the capabilities to handle novel objects in an open-world setting. Inspired by the advances of large pre-trained models, we propose Keypoint Affordance Learning from Imagined Environments (KALIE), which adapts pre-trained Vision Language Models (VLMs) for robotic control in a scalable manner. Instead of directly producing motor commands, KALIE controls the robot by predicting point-based affordance representations based on natural language instructions and visual observations of the scene. The VLM is trained on 2D images with affordances labeled by humans, bypassing the need for training data collected on robotic systems. Through an affordance-aware data synthesis pipeline, KALIE automatically creates massive high-quality training data based on limited example data manually collected by humans. We demonstrate that KALIE can learn to robustly solve new manipulation tasks with unseen objects given only 50 example data points. Compared to baselines using pre-trained VLMs, our approach consistently achieves superior performance.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- Grounding DINO: Marrying DINO with Grounded Pre-training for Open-Set Object Detection2024
- DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset2024
- MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting2024
- Octo: An Open-Source Generalist Robot Policy2024
- AgentTuning: Enabling Generalized Agent Abilities for LLMs2024
- Adding Conditional Control to Text-to-Image Diffusion Models2023
- Scaling Robot Learning with Semantically Imagined Experience2023
- RT-1: Robotics Transformer for Real-World Control at Scale2023
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collectionThe International Journal of Robotics Research, 2017