Prompt Engineering a Prompt Engineer
Preprint
Abstract
Prompt engineering is a challenging yet crucial task for optimizing the performance of large language models on customized tasks. It requires complex reasoning to examine the model's errors, hypothesize what is missing or misleading in the current prompt, and communicate the task with clarity. While recent works indicate that large language models can be meta-prompted to perform automatic prompt engineering, we argue that their potential is limited due to insufficient guidance for complex reasoning in the meta-prompt. We fill this gap by infusing into the meta-prompt three key components: detailed descriptions, context specification, and a step-by-step reasoning template. The resulting method, named PE2, showcases remarkable versatility across diverse language tasks. It finds prompts that outperform "let's think step by step" by 6.3% on MultiArith and 3.1% on GSM8K, and outperforms competitive baselines on counterfactual tasks by 6.9%. Further, we show that PE2 can make targeted prompt edits, rectify erroneous prompts, and induce multi-step plans for complex tasks.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks2024
- Large Language Models Can Self-Improve2023
- Universal Self-Adaptive Prompting2023
- Instruction Induction: From Few Examples to Natural Language Task Descriptions2023
- Better Zero-Shot Reasoning with Self-Adaptive Prompting2023
- GPS: Genetic Prompt Search for Efficient Few-Shot Learning2022
- RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning2022
- How Can We Know What Language Models Know?Transactions of the Association for Computational Linguistics, 2020