Do better search engines really equate to better clinical decisions? If not, why not?
- 18 August 2020
- journal article
- research article
- Published by Wiley in Journal of the Association for Information Science and Technology
- Vol. 72 (2) , 141-155
Abstract
Previous research has found that improved search engine effectiveness—evaluated using a batch‐style approach—does not always translate to significant improvements in user task performance; however, these prior studies focused on simple recall and precision‐based search tasks. We investigated the same relationship, but for realistic, complex search tasks required in clinical decision making. One hundred and nine clinicians and final year medical students answered 16 clinical questions. Although the search engine did improve answer accuracy by 20 percentage points, there was no significant difference when participants used a more effective, state‐of‐the‐art search engine. We also found that the search engine effectiveness difference, identified in the lab, was diminished by around 70% when the search engines were used with real users. Despite the aid of the search engine, half of the clinical questions were answered incorrectly. We further identified the relative contribution of search engine effectiveness to the overall end task success. We found that the ability to interpret documents correctly was a much more important factor impacting task success. If these findings are representative, information retrieval research may need to reorient its emphasis towards helping users to better understand information, rather than just finding it for them.Keywords
This publication has 19 references indexed in Scilit:
- Barriers to evidence‐based medicine: a systematic reviewJournal of Evaluation in Clinical Practice, 2014
- The value of library and information services in patient care: results of a multisite studyJournal of the Medical Library Association, 2013
- Effectiveness of Clinician-selected Electronic Information Resources for Answering Primary Care Physicians' Information NeedsJournal of the American Medical Informatics Association, 2006
- When will information retrieval be "good enough"?2005
- Do Online Information Retrieval Systems Help Experienced Clinicians Answer Clinical Questions?Journal of the American Medical Informatics Association, 2005
- Factors Associated with Success in Searching MEDLINE and Applying Evidence to Answer Clinical QuestionsJournal of the American Medical Informatics Association, 2002
- Do batch and user evaluations give the same results?2000
- Relevance and retrieval evaluation: Perspectives from medicineJournal of the American Society for Information Science, 1994
- A re-examination of relevance: toward a dynamic, situational definition∗Information Processing & Management, 1990
- RELEVANCE: A review of and a framework for the thinking on the notion in information scienceJournal of the American Society for Information Science, 1975