KALIE: Fine-Tuning Vision-Language Models for Open-World Manipulation Without Robot Data
Authors:
- 19 May 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 9507-9515
Abstract
Building generalist robotic systems involves effectively endowing robots with the capabilities to handle novel objects in an open-world setting. Inspired by the advances of large pre-trained models, we propose Keypoint Affordance Learning from Imagined Environments (KALIE), which adapts pre-trained Vision Language Models (VLMs) for robotic control in a scalable manner. Instead of directly producing motor commands, KALIE controls the robot by predicting pointbased affordance representations based on natural language instructions and visual observations of the scene. The VLM is trained on 2D images with affordances labeled by humans, bypassing the need for training data collected on robotic systems. Through an affordance-aware data synthesis pipeline, KALIE automatically creates massive high-quality training data based on limited example data manually collected by humans. We demonstrate that KALIE can learn to robustly solve new manipulation tasks with unseen objects given only 50 example data points. Compared to baselines using pre-trained VLMs, our approach consistently achieves superior performance.Keywords
All Related Versions
Funding Information
- AFOSR (FA9550-22-1-0273)
- NSF (IIS-2246811)
This publication has 23 references indexed in Scilit:
- Grounding DINO: Marrying DINO with Grounded Pre-training for Open-Set Object Detection2024
- CogAgent: A Visual Language Model for GUI Agents2024
- Adding Conditional Control to Text-to-Image Diffusion Models2023
- NeuralField-LDM: Scene Generation with Hierarchical Latent Diffusion Models2023
- BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Fake it till you make it: face analysis in the wild using synthetic data alone2021
- DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort2021
- Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data Generation2020
- Meta-Sim: Learning to Generate Synthetic Datasets2019