TidyBot: Personalized Robot Assistance with Large Language Models
Authors:
- 1 October 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 3546-3553
Abstract
For a robot to personalize physical assistance effectively, it must learn user preferences that can be generally reapplied to future scenarios. In this work, we investigate personalization of household cleanup with robots that can tidy up rooms by picking up objects and putting them away. A key challenge is determining the proper place to put each object, as people's preferences can vary greatly depending on personal taste or cultural background. For instance, one person may prefer storing shirts in the drawer, while another may prefer them on the shelf. We aim to build systems that can learn such preferences from just a handful of examples via prior interactions with a particular person. We show that robots can combine language-based planning and perception with the few-shot summarization capabilities of large language models (LLMs) to infer generalized user preferences that are broadly applicable to future interactions. This approach enables fast adaptation and achieves 91.2% accuracy on unseen objects in our benchmark dataset. We also demonstrate our approach on a real-world mobile manipulator called TidyBot, which successfully puts away 85.0% of objects in real-world test scenarios.Keywords
All Related Versions
This publication has 29 references indexed in Scilit:
- Active Sorting – An Efficient Training of a Sorting Robot with Active Learning Techniques2018
- VirtualHome: Simulating Household Activities Via Programs2018
- Automated task planning using object arrangement optimization2018
- Classifying and sorting cluttered piles of unknown objects with robots: A learning approach2016
- Robot, organize my shelves! Tidying up objects by predicting user preferences2015
- Automatic generation and detection of highly reliable fiducial markers under occlusionPattern Recognition, 2014
- Automated colored object sorting application for robotic arms2012
- Using manipulation primitives for brick sorting in clutter2012
- Development and Control of a Holonomic Mobile Robot for Mobile Manipulation TasksThe International Journal of Robotics Research, 2000
- WordNetCommunications of the ACM, 1995