Beyond Static Evaluation: Rethinking the Assessment of Personalized Agent Adaptability in Information Retrieval
Authors:
- Kirandeep KaurKKKirandeep Kaur
- Preetam Prabhu Srikar DammuPDPreetam Prabhu Srikar Dammu
- Hideo JohoHJHideo Joho
- Chirag ShahCSChirag Shah
Preprint
Abstract
Personalized AI agents are becoming central to modern information retrieval, yet most evaluation methodologies remain static, relying on fixed benchmarks and one-off metrics that fail to reflect how users' needs evolve over time. These limitations hinder our ability to assess whether agents can meaningfully adapt to individuals across dynamic, longitudinal interactions. In this perspective paper, we propose a conceptual lens for rethinking evaluation in adaptive personalization, shifting the focus from static performance snapshots to interaction-aware, evolving assessments. We organize this lens around three core components: (1) persona-based user simulation with temporally evolving preference models; (2) structured elicitation protocols inspired by reference interviews to extract preferences in context; and (3) adaptation-aware evaluation mechanisms that measure how agent behavior improves across sessions and tasks. While recent works have embraced LLM-driven user simulation, we situate this practice within a broader paradigm for evaluating agents over time. To illustrate our ideas, we conduct a case study in e-commerce search using the PersonalWAB dataset. Beyond presenting a framework, our work lays a conceptual foundation for understanding and evaluating personalization as a continuous, user-centric endeavor.Keywords
All Related Versions
- Version 1, 2025-10-05, ArXiv
- Published version: , 396.
This publication has 18 references indexed in Scilit:
- Large Language Models Empowered Personalized Web Agents2025
- Optimization Methods for Personalizing Large Language Models through Retrieval Augmentation2024
- Knowledge-Augmented Large Language Models for Personalized Contextual Query Suggestion2024
- Recommender Systems in the Era of Large Language Models (LLMs)IEEE Transactions on Knowledge and Data Engineering, 2024
- Conversational Information Seeking2022
- A Theoretical Framework for Conversational Search2017
- Modeling the impact of short- and long-term behavior on search personalization2012
- Content-based Recommender Systems: State of the Art and Trends2010
- Matrix Factorization Techniques for Recommender SystemsComputer, 2009
- The CRANFIELD TESTS ON INDEX LANGUAGE DEVICESAslib Proceedings, 1967