KALM: Keypoint Abstraction Using Large Models for Object-Relative Imitation Learning
Authors:
- 19 May 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 8307-8314
Abstract
Generalization to novel object configurations and instances across diverse tasks and environments is a critical challenge in robotics. Keypoint-based representations have been proven effective as a succinct representation for capturing essential object features, and for establishing a reference frame in action prediction, enabling data-efficient learning of robot skills. However, their manual design nature and reliance on additional human labels limit their scalability. In this paper, we propose KALM, a framework that leverages large pre-trained vision-language models (LMs) to automatically generate taskrelevant and cross-instance consistent keypoints. KALM distills robust and consistent keypoints across views and objects by generating proposals using LMs and verifies them against a small set of robot demonstration data. Based on the generated keypoints, we can train keypoint-conditioned policy models that predict actions in keypoint-centric frames, enabling robots to generalize effectively across varying object poses, camera views, and object instances with similar functional shapes. Our method demonstrates strong performance in the real world, adapting to different tasks and environments from only a handful of demonstrations while requiring no additional labels. Videos can be found at https://kalm-il.github.io/.Keywords
All Related Versions
Funding Information
- NSF (2214177)
- AFOSR (FA9550-22-1-0249)
- ARO (W911NF-23-1-0034)
This publication has 29 references indexed in Scilit:
- Segment Anything2023
- KPAM: KeyPoint Affordances for Category-Level Robotic Manipulation2022
- The Functional Correspondence ProblemProceedings of SPIE, 2021
- KETO: Learning Keypoint Representations for Tool Manipulation2020
- Deep Object-Centric Representations for Generalizable Robot Learning2018
- Part-based grasp planning for familiar objects2016
- Affordance detection of tool parts from geometric features2015
- Generalizing pouring actions between objects using warped parameters2014
- Manipulation planning of similar objects by part correspondence2011
- The farthest point strategy for progressive image samplingIEEE Transactions on Image Processing, 1997