Performance of recommender algorithms on top-n recommendation tasks
Authors:
Top Cited Papers
Open Access
- 26 September 2010
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 39-46
Abstract
In many commercial systems, the 'best bet' recommendations are shown, but the predicted rating values are not. This is usually referred to as a top-N recommendation task, where the goal of the recommender system is to find a few specific items which are supposed to be most appealing to the user. Common methodologies based on error metrics (such as RMSE) are not a natural fit for evaluating the top-N recommendation task. Rather, top-N performance can be directly measured by alternative methodologies based on accuracy metrics (such as precision/recall). An extensive evaluation of several state-of-the art recommender algorithms suggests that algorithms optimized for minimizing RMSE do not necessarily perform as expected in terms of top-N recommendation task. Results show that improvements in RMSE often do not translate into accuracy improvements. In particular, a naive non-personalized algorithm can outperform some common recommendation approaches and almost match the accuracy of sophisticated algorithms. Another finding is that the very few top popular items can skew the top-N performance. The analysis points out that when evaluating a recommender algorithm on the top-N recommendation task, the test set should be chosen carefully in order to not bias accuracy metrics towards non-personalized solutions. Finally, we offer practitioners new variants of two collaborative filtering algorithms that, regardless of their RMSE, significantly outperform other recommender algorithms in pursuing the top-N recommendation task, with offering additional practical advantages. This comes at surprise given the simplicity of these two methodsKeywords
This publication has 11 references indexed in Scilit:
- Matrix Factorization Techniques for Recommender SystemsComputer, 2009
- Collaborative filtering with temporal dynamics2009
- An Evaluation Methodology for Collaborative Recommender Systems2008
- Factorization meets the neighborhood2008
- Item-based top-Nrecommendation algorithmsACM Transactions on Information Systems, 2004
- Evaluating collaborative filtering recommender systemsACM Transactions on Information Systems, 2004
- MovieLens unplugged2003
- Item-based collaborative filtering recommendation algorithms2001
- Application of Dimensionality Reduction in Recommender System - A Case Study2000
- Large-Scale Sparse Singular Value ComputationsThe International Journal of Supercomputing Applications, 1992