RLHF Fine-Tuning of LLMs for Alignment with Implicit User Feedback in Conversational Recommenders
Authors:
- 8 August 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 587-591
Abstract
Conversational recommender systems (CRS) based on Large Language Models (LLMs) need to constantly be aligned to the user preferences to provide satisfying and context-relevant item recommendations. The traditional supervised fine-tuning cannot capture the implicit feedback signal, e.g., dwell time, sentiment polarity, or engagement patterns. In this paper, we share a fine-tuning solution using human feedback reinforcement learning (RLHF) to maximize implied user feedback (IUF) in a multi-turn recommendation context. We specify a reward model $R_{-} \varphi$ learnt on weakly-labelled engagement information and maximize user-centric utility by optimizing the foundational LLM $M_{-} \theta$ through a proximal policy optimization (PPO) approach. The architecture models conversational state transitions $s_{-} t \rightarrow a_{-} t \rightarrow s_{-}\{t+1\}$, where the action a_t is associated with LLM-generated item suggestions only on condition of conversation history in the past. The evaluation across synthetic and real-world datasets (e.g.REDIAL, OpenDialKG) demonstrates that our RLHF-fine-tuned models can perform better in terms of top-k recommendation accuracy, coherence, and user satisfaction compared to (arrow-zero-cmwrquca-teja- falset ensuite 2Round group-deca States penalty give up This paper shows that implicit signal alignment can be efficient in achieving scalable and user-adaptive design of CRS.
Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMsACM Computing Surveys, 2025
- LLM-Powered Topic Modeling for Discovering Public Mental Health Trends in Social Media2025
- Research on Personalized Financial Product Recommendation by Integrating Large Language Models and Graph Neural Networks2025
- Revolutionizing Drug Discovery: Integrating Spatial Transcriptomics with Advanced Computer Vision Techniques2025
- Review-based Recommender Systems: A Survey of Approaches, Challenges and Future PerspectivesACM Computing Surveys, 2025
- Exploring time series analysis in frequency domain with complex-valued spectral attention and bidirectional variable mambaThe Journal of Supercomputing, 2025
- Research on the Design of a Short Video Recommendation System Based on Multimodal Information and Differential Privacy2025
- Optimized Credit Score Prediction via an Ensemble Model and SMOTEENN Integration2024
- A Multimodal Fusion Network for Student Emotion Recognition Based on Transformer and Tensor Product2024
- Proximal Policy Optimization With Policy FeedbackIEEE Transactions on Systems, Man, and Cybernetics: Systems, 2021