Discrete Prompt Compression With Reinforcement Learning
Open Access
- 20 May 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 12, 72578-72587
Abstract
Compressed prompts aid instruction-tuned language models (LMs) in overcoming context window limitations and reducing computational costs. Existing methods, which primarily based on training embeddings, face various challenges associated with interpretability, the fixed number of embedding tokens, reusability across different LMs, and inapplicability when interacting with black-box APIs. This study proposes prompt compression with reinforcement learning (PCRL), which is a discrete prompt compression method that addresses these issues. The proposed PCRL method utilizes a computationally efficient policy network that edits prompts directly. The training approach employed in the proposed PCRLs can be applied flexibly to various types of LMs, including both decoder-only and encoder-decoder architecture and it can be trained without gradient access to the LMs or labeled data. The proposed PCRL achieves an average reduction of 24.6% in terms of the token count across various instruction prompts while maintaining sufficient performance. In addition, we demonstrate that the learned policy can be transferred to larger LMs, and through a comprehensive analysis, we explore the token importance within the prompts.Keywords
All Related Versions
Funding Information
- Culture, Sports and Tourism R&D Program through the Korea Creative Content Agency (R2022020070)
- Ministry of Culture, Sports and Tourism in 2022 (Project Name: Development of artificial intelligence-based game simulation technology to support online game content production (R2022020070)
This publication has 20 references indexed in Scilit:
- ChatGPT outperforms crowd workers for text-annotation tasksProceedings of the National Academy of Sciences of the United States of America, 2023
- Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech2023
- Efficient Unsupervised Sentence Compression by Fine-tuning Transformers with Reinforcement Learning2022
- The Power of Scale for Parameter-Efficient Prompt Tuning2021
- Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference2021
- Keep It Simple: Unsupervised Simplification of Multi-Paragraph Text2021
- AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts2020
- The Summary Loop: Learning to Write Abstractive Summaries Without Examples2020
- Discrete Optimization for Unsupervised Sentence Summarization with Word-Level Extraction2020
- Self-Critical Sequence Training for Image Captioning2017