A model for quantitative evaluation of an end‐to‐end question‐answering system
Authors:
- 23 April 2007
- journal article
- research article
- Published by Wiley in Journal of the American Society for Information Science and Technology
- Vol. 58 (8) , 1082-1099
Abstract
We describe a procedure for quantitative evaluation of interactive question‐answering systems and illustrate it with application to the High‐Quality Interactive Question‐Answering (HITIQA) system. Our objectives were (a) to design a method to realistically and reliably assess interactive question‐answering systems by comparing the quality of reports produced using different systems, (b) to conduct a pilot test of this method, and (c) to perform a formative evaluation of the HITIQA system. Far more important than the specific information gathered from this pilot evaluation is the development of (a) a protocol for evaluating an emerging technology, (b) reusable assessment instruments, and (c) the knowledge gained in conducting the evaluation. We conclude that this method, which uses a surprisingly small number of subjects and does not rely on predetermined relevance judgments, measures the impact of system change on work produced by users. Therefore this method can be used to compare the product of interactive systems that use different underlying technologies.Keywords
This publication has 30 references indexed in Scilit:
- Using interview data to identify evaluation criteria for interactive, analytical question-answering systemsJournal of the American Society for Information Science and Technology, 2007
- Cross‐Evaluation: A new model for information system evaluationJournal of the American Society for Information Science and Technology, 2006
- The concept of relevance in IRJournal of the American Society for Information Science and Technology, 2003
- HITIQA2003
- The TREC interactive track: an annotated bibliographyInformation Processing & Management, 2001
- Changes in relevance criteria and problem stages in task performanceJournal of Documentation, 2000
- Do batch and user evaluations give the same results?2000
- Real life, real users, and real needs: a study and analysis of user queries on the webInformation Processing & Management, 2000
- The development of a method for the evaluation of interactive information retrieval systemsJournal of Documentation, 1997
- Multiple search sessions model of end-user behavior: An exploratory studyJournal of the American Society for Information Science, 1996