Universal Self-Adaptive Prompting
Preprint
Abstract
A hallmark of modern large language models (LLMs) is their impressive general zero-shot and few-shot abilities, often elicited through in-context learning (ICL) via prompting. However, while highly coveted and being the most general, zero-shot performances in LLMs are still typically weaker due to the lack of guidance and the difficulty of applying existing automatic prompt design methods in general tasks when ground-truth labels are unavailable. In this study, we address this by presenting Universal Self-Adaptive Prompting (USP), an automatic prompt design approach specifically tailored for zero-shot learning (while compatible with few-shot). Requiring only a small amount of unlabeled data and an inference-only LLM, USP is highly versatile: to achieve universal prompting, USP categorizes a possible NLP task into one of the three possible task types and then uses a corresponding selector to select the most suitable queries and zero-shot model-generated responses as pseudo-demonstrations, thereby generalizing ICL to the zero-shot setup in a fully automated way. We evaluate USP with PaLM and PaLM 2 models and demonstrate performances that are considerably stronger than standard zero-shot baselines and often comparable to or even superior to few-shot baselines across more than 40 natural language understanding, natural language generation, and reasoning tasks.Keywords
All Related Versions
This publication has 24 references indexed in Scilit:
- Reframing Instructional Prompts to GPTk’s Language2022
- Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models2022
- Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models2022
- What Makes Good In-Context Examples for GPT-3?2022
- The Power of Scale for Parameter-Efficient Prompt Tuning2021
- Prefix-Tuning: Optimizing Continuous Prompts for Generation2021
- WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarization2020
- Text Classification Using Label Names Only: A Language Model Self-Training Approach2020
- Adversarial NLI: A New Benchmark for Natural Language Understanding2020
- Natural Questions: A Benchmark for Question Answering ResearchTransactions of the Association for Computational Linguistics, 2019