From Variability to Stability: Advancing RecSys Benchmarking Practices
Authors:
Open Access
- 24 August 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 5701-5712
Abstract
In the rapidly evolving domain of Recommender Systems (RecSys), new algorithms frequently claim state-of-the-art performance based on evaluations over a limited set of arbitrarily selected datasets. However, this approach may fail to holistically reflect their effectiveness due to the significant impact of dataset characteristics on algorithm performance. Addressing this deficiency, this paper introduces a novel benchmarking methodology to facilitate a fair and robust comparison of RecSys algorithms, thereby advancing evaluation practices. By utilizing a diverse set of 30 open datasets, including two introduced in this work, and evaluating 11 collaborative filtering algorithms across 9 metrics, we critically examine the influence of dataset characteristics on algorithm performance. We further investigate the feasibility of aggregating outcomes from multiple datasets into a unified ranking. Through rigorous experimental analysis, we validate the reliability of our methodology under the variability of datasets, offering a benchmarking strategy that balances quality and computational demands. This methodology enables a fair yet effective means of evaluating RecSys algorithms, providing valuable guidance for future research endeavors.Keywords
All Related Versions
Funding Information
- the Analytical center under the RF Government (Subsidy agreement 000000D730321P5Q0002, Grant No. 70-2021-00145 02.11.2021)
This publication has 40 references indexed in Scilit:
- Offline recommender system evaluation: Challenges and new directionsAI Magazine, 2022
- Over‐optimism in benchmark studies and the multiplicity of design and analysis options when interpreting their resultsWIREs Data Mining and Knowledge Discovery, 2021
- HIVE-COTE 2.0: a new meta ensemble for time series classificationMachine Learning, 2021
- Explaining recommender systems fairness and accuracy through the lens of data characteristicsInformation Processing & Management, 2021
- DeepRec: A deep neural network approach to recommendation with item embedding and weighted loss functionInformation Sciences, 2019
- Statistical biases in Information Retrieval metrics for recommender systemsInformation Retrieval Journal, 2017
- GTApprox: Surrogate modeling for industrial designAdvances in Engineering Software, 2016
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Time-aware recommender systems: a comprehensive survey and analysis of existing evaluation protocolsUser Modeling and User-Adapted Interaction, 2013
- Matrix Factorization Techniques for Recommender SystemsComputer, 2009