Abstract
A result page of a modern search engine often goes beyond a simple list of “10 blue links.” Many specific user needs (e.g., News, Image, Video) are addressed by so-called aggregated or vertical search solutions: specially presented documents, often retrieved from specific sources, that stand out from the regular organic Web search results. When it comes to evaluating ranking systems, such complex result layouts raise their own challenges. This is especially true for so-called interleaving methods that have arisen as an important type of online evaluation: by mixing results from two different result pages, interleaving can easily break the desired Web layout in which vertical documents are grouped together, and hence hurt the user experience. We conduct an analysis of different interleaving methods as applied to aggregated search engine result pages. Apart from conventional interleaving methods, we propose two vertical-aware methods: one derived from the widely used Team-Draft Interleaving method by adjusting it in such a way that it respects vertical document groupings, and another based on the recently introduced Optimized Interleaving framework. We show that our proposed methods are better at preserving the user experience than existing interleaving methods while still performing well as a tool for comparing ranking systems. For evaluating our proposed vertical-aware interleaving methods, we use real-world click data as well as simulated clicks and simulated ranking systems.
Funding Information
  • Dutch national program COMMIT
  • Microsoft Research Ph.D. program
  • The Netherlands Organisation for Scientific Research (NWO) (727.011.005, 612.001.116, HOR-11-10, and 640.006.013)
  • Yahoo! Faculty Research and Engagement Program
  • European Community's Seventh Framework Programme (FP7/2007-2013) (288024 (LiMoSINe) and 312827 (VOX-Pol))
  • the Netherlands eScience Center under project number 027.012.105
  • Elite Network Shifts project funded by the Royal Dutch Academy of Sciences (KNAW)
  • Center for Creation, Content and Technology (CCCT)
  • TROVe project funded by the CLARIAH program
  • HPC Fund
  • QuaMerdes project funded by the CLARIN-nl program
  • ESF Research Network Program ELIAS

This publication has 32 references indexed in Scilit: