On-Robot Reinforcement Learning with Goal-Contrastive Rewards
Authors:
- 19 May 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 4797-4805
Abstract
Reinforcement Learning (RL) has the potential to enable robots to learn from their own actions in the real world. Unfortunately, RL can be prohibitively expensive, in terms of on-robot runtime, due to inefficient exploration when learning from a sparse reward signal. Designing dense reward functions is labour-intensive and requires domain expertise. In our work, we propose Goal-Contrastive Rewards (GCR), a dense reward function learning method that can be trained on passive video demonstrations. By using videos without actions, our method is easier to scale, as we can use arbitrary videos. GCR combines two loss functions, an implicit value loss function that models how the reward increases when traversing a successful trajectory, and a goal-contrastive loss that discriminates between successful and failed trajectories. We perform experiments in simulated manipulation environments across RoboMimic and MimicGen tasks, as well as in the real world using a Franka arm and a Spot quadruped. We find that GCR leads to a more-sample efficient RL, enabling model-free RL to solve about twice as many tasks as our baseline reward learning methods. We also demonstrate positive cross-embodiment transfer from videos of people and of other robots performing a task. Website: https://gcr-robot.github.io/.Keywords
All Related Versions
Funding Information
- NSF (1750649,2107256,2314182,2409351)
This publication has 24 references indexed in Scilit:
- Grounding DINO: Marrying DINO with Grounded Pre-training for Open-Set Object Detection2024
- EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World2024
- Segment Anything2023
- Affordances from Human Videos as a Versatile Representation for Robotics2023
- DexMV: Imitation Learning for Dexterous Manipulation from Human Videos2022
- Ego4D: Around the World in 3,000 Hours of Egocentric Video2022
- Human Hands as Probes for Interactive Object Understanding2022
- Learning Predictive Models from Observation and Interaction2020
- Demo2Vec: Reasoning Object Affordances from Online Videos2018
- Time-Contrastive Networks: Self-Supervised Learning from Video2018