Bootstrapping Linear Models for Fast Online Adaptation in Human-Agent Collaboration
Preprint
Abstract
Agents that assist people need to have well-initialized policies that can adapt quickly to align with their partners' reward functions. Initializing policies to maximize performance with unknown partners can be achieved by bootstrapping nonlinear models using imitation learning over large, offline datasets. Such policies can require prohibitive computation to fine-tune in-situ and therefore may miss critical run-time information about a partner's reward function as expressed through their immediate behavior. In contrast, online logistic regression using low-capacity models performs rapid inference and fine-tuning updates and thus can make effective use of immediate in-task behavior for reward function alignment. However, these low-capacity models cannot be bootstrapped as effectively by offline datasets and thus have poor initializations. We propose BLR-HAC, Bootstrapped Logistic Regression for Human Agent Collaboration, which bootstraps large nonlinear models to learn the parameters of a low-capacity model which then uses online logistic regression for updates during collaboration. We test BLR-HAC in a simulated surface rearrangement task and demonstrate that it achieves higher zero-shot accuracy than shallow methods and takes far less computation to adapt online while still achieving similar performance to fine-tuned, large nonlinear models. For code, please see our project page https://sites.google.com/view/blr-hac.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Towards Online Adaptation for Autonomous Household Assistants2023
- Android as a Receptionist in a Shopping Mall Using Inverse Reinforcement LearningIEEE Robotics and Automation Letters, 2022
- ASHA: Assistive Teleoperation via Human-in-the-Loop Reinforcement Learning2022
- Introducing CARESSER: A framework for in situ learning robot social assistance from expert knowledge and demonstrationsUser Modeling and User-Adapted Interaction, 2022
- HARMONIC: A multimodal dataset of assistive human–robot collaborationThe International Journal of Robotics Research, 2021
- Examining the Effects of Anticipatory Robot Assistance on Human Decision Making2020
- Evaluating Fluency in Human–Robot CollaborationIEEE Transactions on Human-Machine Systems, 2019
- Shared autonomy via hindsight optimization for teleoperation and teamingThe International Journal of Robotics Research, 2018
- Eye-Hand Behavior in Human-Robot Shared Manipulation2018
- Mobile manipulation through an assistive home robot2012