Dual-Actor Fine-Tuning of VLA Models: A Talk-and-Tweak Human-in-the-Loop Approach
Authors:
- Piaopiao JinPJPiaopiao Jin
- Qi WangQWQi Wang
- Guokang SunGSGuokang Sun
- Ziwen CaiZCZiwen Cai
- Pinjia HePHPinjia He
- Yangwei YouYYYangwei You
Preprint
Abstract
Vision-language-action (VLA) models demonstrate strong generalization in robotic manipulation but face challenges in complex, real-world tasks. While supervised fine-tuning with demonstrations is constrained by data quality, reinforcement learning (RL) offers a promising alternative. We propose a human-in-the-loop dual-actor fine-tuning framework grounded in RL. The framework integrates a primary actor for robust multi-task performance with a refinement actor for latent-space adaptation. Beyond standard physical interventions, we introduce a lightweight talk-and-tweak scheme that converts human corrections into semantically grounded language commands, thereby generating a new dataset for policy learning. In real-world multi-task experiments, our approach achieves 100% success across three tasks within 101 minutes of online fine-tuning. For long-horizon tasks, it sustains a 50% success rate over 12 consecutive operations. Furthermore, the framework scales effectively to multi-robot training, achieving up to a 2 times improvement in efficiency when using dual robots. The experiment videos are available at https://sites.google.com/view/hil-daft/.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Precise and dexterous robotic manipulation via human-in-the-loop reinforcement learningScience Robotics, 2025
- SRT-H: A hierarchical framework for autonomous surgery via language-conditioned imitation learningScience Robotics, 2025
- ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy2025
- From Imitation to Refinement - Residual Rl for Precise Assembly2025
- FLaRe: Achieving Masterful and Adaptive Robot Policies with Large-Scale Reinforcement Learning Fine-Tuning2025
- Diffusion policy: Visuomotor policy learning via action diffusionThe International Journal of Robotics Research, 2024
- Yell At Your Robot: Improving On-the-Fly from Language Corrections2024
- Training Language Models to Follow Instructions with Human Feedback2022
- HG-DAgger: Interactive Imitation Learning with Human Experts2019
- Deep Residual Learning for Image Recognition2016