AWAlice Wang

Alice Wang

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article



Latest Publications

  • Open Access

    Do LLM-judges Align with Human Relevance in Cranfield-style Recommender Evaluation?

    by Gustavo Penha, Aleksandr V. Petrov, Claudia Hauff, Enrico Palumbo, Ali Vardasbi, Edoardo D'Amico, Francesco Fabbri, Alice Wang, Praveen Chandar, Henrik Lindstrom, et al.

    Evaluating recommender systems remains a long-standing challenge, as offline methods based on historical user interactions and train-test splits often yield unstable and inconsistent results due to exposure bias, popularity bias, sampled evaluations, and missing-not-at-random patterns. In contrast, textual document retrieval benefits from robust, standardized evaluation via Cranfield-style test collections, which combine pooled relevance judgments with controlled setups. While recent work shows that adapting this methodology to recommender systems is feasible, constructing such collections remains costly due to the need for manual relevance judgments, thus limiting scalability. This paper investigates whether Large Language Models (LLMs) can serve as reliable automatic judges to address these scalability challenges. Using the ML-32M-ext Cranfield-style movie recommendation collection, we first examine the limitations of existing evaluation methodologies. Then we explore the alignment and the recommender systems ranking agreement between the LLM-judge and human provided relevance labels. We find that incorporating richer item metadata and longer user histories improves alignment, and that LLM-judge yields high agreement with human-based rankings (Kendall's tau = 0.87). Finally, an industrial case study in the podcast recommendation domain demonstrates the practical value of LLM-judge for model selection. Overall, our results show that LLM-judge is a viable and scalable approach for evaluating recommender systems.


  • Open Access

    Contextualizing Spotify's Audiobook List Recommendations with Descriptive Shelves

    by Gustavo Penha, Alice Wang, Martin Achenbach, Kristen Sheets, Sahitya Mantravadi, Remi Galvez, Nico Guetta-Jeanrenaud, Divya Narayanan, Ofeliya Kalaydzhyan, Hugues Bouchard

    In this paper, we propose a pipeline to generate contextualized list recommendations with descriptive shelves in the domain of audiobooks. By creating several shelves for topics the user has an affinity to, e.g. Uplifting Women's Fiction, we can help them explore their recommendations according to their interests and at the same time recommend a diverse set of items. To do so, we use Large Language Models (LLMs) to enrich each item's metadata based on a taxonomy created for this domain. Then we create diverse descriptive shelves for each user. A/B tests show improvements in user engagement and audiobook discovery metrics, demonstrating benefits for users and content creators.

  • Contextualizing Spotify’s Audiobook List Recommendations with Descriptive Shelves

    by Gustavo Penha, Alice Wang, Martin Achenbach, Kristen Sheets, Sahitya Mantravadi, Remi Galvez, Nico Guetta-Jeanrenaud, Divya Narayanan, Ofeliya Kalaydzhyan, Hugues Bouchard