Improving Generalization of Alignment with Human Preferences through Group Invariant Learning
Preprint
Abstract
The success of AI assistants based on language models (LLMs) hinges crucially on Reinforcement Learning from Human Feedback (RLHF), which enables the generation of responses more aligned with human preferences. As universal AI assistants, there's a growing expectation for them to perform consistently across various domains. However, previous work shows that Reinforcement Learning (RL) often exploits shortcuts to attain high rewards and overlooks challenging samples. This focus on quick reward gains undermines both the stability in training and the model's ability to generalize to new, unseen data. In this work, we propose a novel approach that can learn a consistent policy via RL across various data groups or domains. Given the challenges associated with acquiring group annotations, our method automatically classifies data into different groups, deliberately maximizing performance variance. Then, we optimize the policy to perform well on challenging groups. Lastly, leveraging the established groups, our approach adaptively adjusts the exploration space, allocating more learning capacity to more challenging data and preventing the model from over-optimizing on simpler data. Experimental results indicate that our approach significantly enhances training stability and model generalization.Keywords
All Related Versions
This publication has 5 references indexed in Scilit:
- Reward Collapse in Aligning Large Language ModelsJournal of Data Science, 2025
- Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Study2023
- On the Dangers of Stochastic Parrots2021
- ZeRO: Memory optimizations Toward Training Trillion Parameter Models2020
- X. On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random samplingJournal of Computers in Education, 1900