Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs
Preprint
Abstract
A popular approach for improving the correctness of output from large language models (LLMs) is Self-Consistency - poll the LLM multiple times and output the most frequent solution. Existing Self-Consistency techniques always generate a constant number of samples per question, where a better approach will be to non-uniformly distribute the available budget based on the amount of agreement in the samples generated so far. In response, we introduce Adaptive-Consistency, a cost-efficient, model-agnostic technique that dynamically adjusts the number of samples per question using a lightweight stopping criterion. Our experiments over 17 reasoning and code generation datasets and three LLMs demonstrate that Adaptive-Consistency reduces sample budget by up to 7.9 times with an average accuracy drop of less than 0.1%. Our code and data are available at https://www.sample-step-by-step.infoKeywords
All Related Versions
This publication has 4 references indexed in Scilit:
- Did Aristotle Use a Laptop?A Question Answering Benchmark with Implicit Reasoning StrategiesTransactions of the Association for Computational Linguistics, 2021
- A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers2020
- Crowdsourcing systems on the World-Wide WebCommunications of the ACM, 2011
- Adaptive Importance Sampling to Accelerate Training of a Neural Probabilistic Language ModelIEEE Transactions on Neural Networks, 2008