VORTEX: Aligning Task Utility and Human Preferences through LLM-Guided Reward Shaping
Preprint
Abstract
In social impact optimization, AI decision systems often rely on solvers that optimize well-calibrated mathematical objectives. However, these solvers cannot directly accommodate evolving human preferences, typically expressed in natural language rather than formal constraints. Recent approaches address this by using large language models (LLMs) to generate new reward functions from preference descriptions. While flexible, they risk sacrificing the system's core utility guarantees. In this paper, we propose \texttt{VORTEX}, a language-guided reward shaping framework that preserves established optimization goals while adaptively incorporating human feedback. By formalizing the problem as multi-objective optimization, we use LLMs to iteratively generate shaping rewards based on verbal reinforcement and text-gradient prompt updates. This allows stakeholders to steer decision behavior via natural language without modifying solvers or specifying trade-off weights. We provide theoretical guarantees that \texttt{VORTEX} converges to Pareto-optimal trade-offs between utility and preference satisfaction. Empirical results in real-world allocation tasks demonstrate that \texttt{VORTEX} outperforms baselines in satisfying human-aligned coverage goals while maintaining high task performance. This work introduces a practical and theoretically grounded paradigm for human-AI collaborative optimization guided by natural language.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Balancing Act: Prioritization Strategies for LLM-Designed Restless Bandit Rewards2025
- Finite-Horizon Single-Pull Restless Bandits: An Efficient Index Policy For Scarce Resource Allocation2025
- FLAG-TRADER: Fusion LLM-Agent with Gradient-based Reinforcement Learning for Financial Trading2025
- FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Making2024
- A Decision-Language Model (DLM) for Dynamic Restless Multi-Armed Bandit Tasks in Public Health2024
- Learning Infinite-Horizon Average-Reward Restless Multi-Action Bandits Via Index Awareness2022
- A snapshot of the frontiers of fairness in machine learningCommunications of the ACM, 2020
- Dissecting racial bias in an algorithm used to manage the health of populationsScience, 2019
- Trade‐offs and efficiencies in optimal budget‐constrained multispecies corridor networksConservation Biology, 2016
- Optimized resource allocation for emergency response after earthquake disastersSafety Science, 2000