What Else Do I Need to Know? The Effect of Background Information on Users' Reliance on QA Systems
Preprint
Abstract
NLP systems have shown impressive performance at answering questions by retrieving relevant context. However, with the increasingly large models, it is impossible and often undesirable to constrain models' knowledge or reasoning to only the retrieved context. This leads to a mismatch between the information that the models access to derive the answer and the information that is available to the user to assess the model predicted answer. In this work, we study how users interact with QA systems in the absence of sufficient information to assess their predictions. Further, we ask whether adding the requisite background helps mitigate users' over-reliance on predictions. Our study reveals that users rely on model predictions even in the absence of sufficient information needed to assess the model's correctness. Providing the relevant background, however, helps users better catch model errors, reducing over-reliance on incorrect predictions. On the flip side, background information also increases users' confidence in their accurate as well as inaccurate judgments. Our work highlights that supporting users' verification of QA predictions is an important, yet challenging, problem.Keywords
All Related Versions
This publication has 33 references indexed in Scilit:
- In search of verifiability: Explanations rarely enable complementary performance in AI‐advised decision makingAI Magazine, 2024
- Do as AI say: susceptibility in deployment of clinical decision-aidsnpj Digital Medicine, 2021
- How machine-learning recommendations influence clinician treatment selections: the example of antidepressant selectionTranslational Psychiatry, 2021
- The challenge of crafting intelligible intelligenceCommunications of the ACM, 2019
- Agency plus automation: Designing artificial intelligence into interactive systemsProceedings of the National Academy of Sciences of the United States of America, 2019
- Disparate Interactions2019
- The Mythos of Model InterpretabilityQueue, 2018
- "Why Should I Trust You?"2016
- The Role of Explanations on Trust and Reliance in Clinical Decision Support Systems2015
- Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple TestingJournal of the Royal Statistical Society Series B: Statistical Methodology, 1995