Designing Formulae for Ranking Search Results: Mixed Methods Evaluation Study (Preprint)
Preprint
- 11 May 2021
- preprint
- Published by JMIR Publications Inc. in JMIR Preprints
Abstract
BACKGROUND A major factor in the success of any search engine is the relevance of the search results; a tool should sort the search results to present the most relevant documents first. Assessing the performance of the ranking formula is an important part of search engine evaluation. However, the methods currently used to evaluate ranking formulae mainly collect quantitative data and do not gather qualitative data, which help to understand what needs to be improved to tailor the formulae to their end users. OBJECTIVE This study aims to evaluate 2 different parameter settings of the ranking formula of LiSSa (the French acronym for scientific literature in health care; Department of Medical Informatics and Information), a tool that provides access to health scientific literature in French, to adapt the formula to the needs of the end users. METHODS To collect quantitative and qualitative data, user tests were carried out with representative end users of LiSSa: 10 general practitioners and 10 registrars. Participants first assessed the relevance of the search results and then rated the ranking criteria used in the 2 formulae. Verbalizations were analyzed to characterize each criterion. RESULTS A formula that prioritized articles representing a consensus in the field was preferred. When users assess an article’s relevance, they judge its topic, methods, and value in clinical practice. CONCLUSIONS Following the evaluation, several improvements were implemented to give more weight to articles that match the search topic and to downgrade articles that have less informative or scientific value for the reader. Applying a qualitative methodology generates valuable user inputs to improve the ranking formula and move toward a highly usable search engine.Keywords
All Related Versions
- Published version: JMIR Human Factors, 9 (1), e30258.
This publication has 30 references indexed in Scilit:
- Perceptions of document relevanceFrontiers in Psychology, 2014
- A systematic review of interactive information retrieval evaluation studies, 1967–2006Journal of the American Society for Information Science and Technology, 2013
- A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidenceInternational Journal of Medical Informatics, 2009
- Multiple Terminologies in a Health Portal: Automatic Indexing and Information Retrieval2009
- On the history of evaluation in IRJournal of Information Science, 2008
- Relevance: A review of the literature and a framework for thinking on the notion in information science. Part II: nature and manifestations of relevanceJournal of the American Society for Information Science and Technology, 2007
- TREC: Experiment and evaluation in information retrievalJournal of the American Society for Information Science and Technology, 2007
- Subjective Relevance: Implications on Digital Libraries for Experts and NovicesLecture Notes in Computer Science, 2004
- Cumulated gain-based evaluation of IR techniquesACM Transactions on Information Systems, 2002
- The Philosophy of Information Retrieval Evaluation2002