LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error
Authors:
- Boshi WangBWBoshi Wang
- Hao FangHFHao Fang
- Jason EisnerJEJason Eisner
Preprint
Abstract
Tools are essential for large language models (LLMs) to acquire up-to-date information and take consequential actions in external environments. Existing work on tool-augmented LLMs primarily focuses on the broad coverage of tools and the flexibility of adding new tools. However, a critical aspect that has surprisingly been understudied is simply how accurately an LLM uses tools for which it has been trained. We find that existing LLMs, including GPT-4 and open-source LLMs specifically fine-tuned for tool use, only reach a correctness rate in the range of 30% to 60%, far from reliable use in practice. We propose a biologically inspired method for tool-augmented LLMs, simulated trial and error (STE), that orchestrates three key mechanisms for successful tool use behaviors in the biological system: trial and error, imagination, and memory. Specifically, STE leverages an LLM's 'imagination' to simulate plausible scenarios for using a tool, after which the LLM interacts with the tool to learn from its execution feedback. Both short-term and long-term memory are employed to improve the depth and breadth of the exploration, respectively. Comprehensive experiments on ToolBench show that STE substantially improves tool learning for LLMs under both in-context learning and fine-tuning settings, bringing a boost of 46.7% to Mistral-Instruct-7B and enabling it to outperform GPT-4. We also show effective continual learning of tools via a simple experience replay strategy.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-TuningIEEE Transactions on Audio, Speech and Language Processing, 2025
- Automatically Correcting Large Language Models: Surveying the Landscape of Diverse Automated Correction StrategiesTransactions of the Association for Computational Linguistics, 2024
- ExpeL: LLM Agents Are Experiential LearnersProceedings of the AAAI Conference on Artificial Intelligence, 2024
- MemoryBank: Enhancing Large Language Models with Long-Term MemoryProceedings of the AAAI Conference on Artificial Intelligence, 2024
- Learning To Retrieve Prompts for In-Context Learning2022
- What Makes Good In-Context Examples for GPT-3?2022
- Constrained Language Models Yield Few-Shot Semantic Parsers2021
- Generating Datasets with Pretrained Language Models2021
- Overcoming catastrophic forgetting in neural networksProceedings of the National Academy of Sciences of the United States of America, 2017
- The cognitive bases of human tool useBehavioral and Brain Sciences, 2012