TidyBot: Personalized Robot Assistance with Large Language Models
Preprint
Abstract
For a robot to personalize physical assistance effectively, it must learn user preferences that can be generally reapplied to future scenarios. In this work, we investigate personalization of household cleanup with robots that can tidy up rooms by picking up objects and putting them away. A key challenge is determining the proper place to put each object, as people's preferences can vary greatly depending on personal taste or cultural background. For instance, one person may prefer storing shirts in the drawer, while another may prefer them on the shelf. We aim to build systems that can learn such preferences from just a handful of examples via prior interactions with a particular person. We show that robots can combine language-based planning and perception with the few-shot summarization capabilities of large language models (LLMs) to infer generalized user preferences that are broadly applicable to future interactions. This approach enables fast adaptation and achieves 91.2% accuracy on unseen objects in our benchmark dataset. We also demonstrate our approach on a real-world mobile manipulator called TidyBot, which successfully puts away 85.0% of objects in real-world test scenarios.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Text2Motion: from natural language instructions to feasible plansAutonomous Robots, 2023
- ProgPrompt: Generating Situated Robot Task Plans using Large Language Models2023
- Grounding Language with Visual Affordances over Unstructured Data2023
- Open-vocabulary Queryable Scene Representations for Real World Planning2023
- Code as Policies: Language Model Programs for Embodied Control2023
- Housekeep: Tidying Virtual Households Using Commonsense Reasoning2022
- Autonomous planning based on spatial concepts to tidy up home environments with service robotsAdvanced Robotics, 2021
- TossingBot: Learning to Throw Arbitrary Objects With Residual PhysicsIEEE Transactions on Robotics, 2020
- Active Sorting – An Efficient Training of a Sorting Robot with Active Learning Techniques2018
- Automatic generation and detection of highly reliable fiducial markers under occlusionPattern Recognition, 2014