A comparative analysis of offline and online evaluations and discussion of research paper recommender system evaluation
Open Access
- 12 October 2013
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 7-14
Abstract
Offline evaluations are the most common evaluation method for research paper recommender systems. However, no thorough discussion on the appropriateness of offline evaluations has taken place, despite some voiced criticism. We conducted a study in which we evaluated various recommendation approaches with both offline and online evaluations. We found that results of offline and online evaluations often contradict each other. We discuss this finding in detail and conclude that offline evaluations may be inappropriate for evaluating research paper recommender systems, in many settings.Keywords
This publication has 20 references indexed in Scilit:
- Introducing Docear's research paper recommender system2013
- The Impact of Demographics (Age and Gender) and Other User-Characteristics on Evaluating Recommender Systems2013
- Explaining the user experience of recommender systemsUser Modeling and User-Adapted Interaction, 2012
- Recommender systems: from algorithms to user experienceUser Modeling and User-Adapted Interaction, 2012
- Evaluating Recommendation Systems2010
- I Like It... I Like It Not: Evaluating User Ratings Noise in Recommender Systems2009
- Evaluating collaborative filtering recommender systemsACM Transactions on Information Systems, 2004
- Do batch and user evaluations give the same results?2000
- PROGRESS IN DOCUMENTATION THE COMPLEXITIES OF CITATION PRACTICE: A REVIEW OF CITATION STUDIESJournal of Documentation, 1993
- Private acts and public objects: An investigation of citer motivationsJournal of the American Society for Information Science, 1985