Preference-Guided Reflective Sampling for Aligning Language Models
Preprint
Abstract
Large language models (LLMs) are aligned with human preferences by reinforcement learning from human feedback (RLHF). Effective data sampling is crucial for RLHF, as it determines the efficiency of model training, ensuring that models learn from the informative samples. To achieve better data generation, we propose a new sampling method called Preference-Guided Reflective Sampling (PRS). PRS frames the response generation as an optimization process to the explicitly specified user preference described in natural language. It employs a tree-based generation framework to enable an efficient sampling process, which guides the direction of generation through preference and better explores the sampling space with adaptive self-refinement. Notably, PRS can align LLMs to diverse preferences. We study preference-controlled text generation for instruction following and keyword-focused document summarization. Our findings indicate that PRS, across different LLM policies, generates training data with much higher rewards than strong baselines. PRS also excels in post-RL training.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- Benchmarking Large Language Models on Controllable Generation under Diversified InstructionsProceedings of the AAAI Conference on Artificial Intelligence, 2024
- From Complex to Simple: Enhancing Multi-Constraint Complex Instruction Following Ability of Large Language Models2024
- Enhancing Chat Language Models by Scaling High-quality Instructional Conversations2023
- Red Teaming Language Models with Language Models2022
- ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection2022
- Get To The Point: Summarization with Pointer-Generator Networks2017