Incorporating Clicks, Attention and Satisfaction into a Search Engine Result Page Evaluation Model
Open Access
- 24 October 2016
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 175-184
Abstract
Modern search engine result pages often provide immediate value to users and organize information in such a way that it is easy to navigate. The core ranking function contributes to this and so do result snippets, smart organization of result blocks and extensive use of one-box answers or side panels. While they are useful to the user and help search engines to stand out, such features present two big challenges for evaluation. First, the presence of such elements on a search engine result page (SERP) may lead to the absence of clicks, which is, however, not related to dissatisfaction, so-called 'good abandonments.' Second, the non-linear layout and visual difference of SERP items may lead to non-trivial patterns of user attention, which is not captured by existing evaluation metrics. In this paper we propose a model of user behavior on a SERP that jointly captures click behavior, user attention and satisfaction, the CAS model, and demonstrate that it gives more accurate predictions of user actions and self-reported satisfaction than existing models based on clicks alone. We use the CAS model to build a novel evaluation metric that can be applied to non-linear SERP layouts and that can account for the utility that users obtain directly on a SERP. We demonstrate that this metric shows better agreement with user-reported satisfaction than conventional evaluation metrics.Keywords
All Related Versions
Funding Information
- the Dutch national program COMMIT
- Bloomberg L.P. (the Bloomberg Research Grant program)
- the European Community's Seventh Framework Programme (FP7/2007-2013) (grant agreement nr 312827 (VOX-Pol))
- the Netherlands eScience Center (project number 027.012.105)
- Yahoo (the Yahoo Faculty Research and Engagement Program)
- Ahold
- Koninklijke Nederlandse Akademie van Wetenschappen (Elite Network Shifts project)
- Yandex
- European Science Foundation (the ESF Research Network Program ELIAS)
- Elsevier
- Amsterdam Data Science
- Microsoft Research (the Microsoft Research Ph.D.program)
- the Netherlands Institute for Sound and Vision
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (project nrs 727.011.005, 612.001.116, HOR-11-10, 640.006.013, 612.066.930, CI-14-25, SH-322-15, 652.002.001, 612.001.551)
This publication has 32 references indexed in Scilit:
- Incorporating Non-sequential Behavior into Click Models2015
- Different Users, Different Opinions2015
- Click Models for Web SearchSynthesis Lectures on Information Concepts, Retrieval, and Services, 2015
- The Three Sides of CrowdTruthHuman Computation, 2014
- Click model-based information retrieval metrics2013
- Differences in search engine evaluations between query owners and non-owners2013
- Interpreting User Inactivity on Search Results2010
- Cumulated gain-based evaluation of IR techniquesACM Transactions on Information Systems, 2002
- Estimating the Reliability, Systematic Error and Random Error of Interval DataEducational and Psychological Measurement, 1970
- A Coefficient of Agreement for Nominal ScalesEducational and Psychological Measurement, 1960