Human-Guided Online Reward Adaptation for Real-Robot Arm Manipulation
- 1 June 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 11 (8) , 9072-9079
Abstract
Real-world reinforcement learning enables robots to adapt directly to complex physical dynamics, avoiding the reality gap inherent in simulation and rendering previously infeasible manipulation skills a reality. However, the high cost of real-world exploration strictly limits sample availability. Such tight interaction budgets make sparse rewards insufficient for sample-efficient learning in real-robot arm manipulation, creating the need for dense reward supervision. Although pretrained reward models provide useful semantic priors, their generic predictions are often misaligned with embodiment-specific and online dynamics during physical deployment. To address this issue, we propose adaptive reward via human interaction (ARHI), a parameter-efficient framework for online adaptation of VLM-based rewards in real-robot reinforcement learning. Rather than treating the pretrained reward as a fixed module, ARHI continuously recalibrates it using lightweight online updates, while preserving the semantic priors of the frozen backbone. To support this online reward adaptation under sparse human feedback, we further integrate the method into an asynchronous dual-loop system that converts sparse interventions into dense supervisory signals for reward and policy learning. Extensive real-world experiments across six manipulation tasks demonstrate that ARHI outperforms strong baselines, reducing human effort by 28.0%, while achieving a 47.6% reduction in performance drop under challenging deployment shifts.Keywords
Funding Information
- National Key R&D Program of China (2024YFB4708900)
- Zhongguancun Academy (C20250301)
This publication has 10 references indexed in Scilit:
- Reward Training Wheels: Adaptive Auxiliary Rewards for Robotics Reinforcement Learning2025
- Precise and dexterous robotic manipulation via human-in-the-loop reinforcement learningScience Robotics, 2025
- On-Robot Reinforcement Learning with Goal-Contrastive Rewards2025
- Diffusion policy: Visuomotor policy learning via action diffusionThe International Journal of Robotics Research, 2024
- Adapt2Reward: Adapting Video-Language Models to Generalizable Robotic Rewards via Failure Prompts2024
- SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learning2024
- GUIDE: Real-Time Human-Shaped Agents2024
- Champion-level drone racing using deep reinforcement learningNature, 2023
- Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware2023
- Daydreamer2007