Emile L. Morse
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Research Interests
Emile L. Morse has a total of 14 co-authors. Here are the top co-authors by number of shared publications:
- Paul B. Kantor
Rutgers University
3 shared publications - Diane Kelly
University of North Carolina at Chapel Hill
2 shared publications - Jean Scholtz
Pacific Northwest National Laboratory, Richland, WA, USA
2 shared publications - Ying Sun
Department of Information Science University at Buffalo Buffalo NY 14260 USA
2 shared publications - Aiping Zhang1 shared publication
Celeste Paul
1 shared publication
Latest Publications
- 5 July 2011
- Published by Wiley in Journal of the American Society for Information Science and Technology
- Vol. 62 (9) , 1653-1665
Using cross-evaluation to evaluate interactive QA systems
- 5 July 2011
- Published by Wiley in Journal of the American Society for Information Science and Technology
- Vol. 62 (9) , 1653-1665
Using cross-evaluation to evaluate interactive QA systems
- 1 January 2009
- Published by Cambridge University Press (CUP) in Natural Language Engineering
- Vol. 15 (1) , 119-141
Questionnaires for eliciting evaluation data from users of interactive question answering systems
Evaluating interactive question answering (QA) systems with real users can be challenging because traditional evaluation measures based on the relevance of items returned are difficult to employ since relevance judgments can be unstable in multi-user evaluations. The work reported in this paper evaluates, in distinguishing among a set of interactive QA systems, the effectiveness of three questionnaires: a Cognitive Workload Questionnaire (NASA TLX), and Task and System Questionnaires customized to a specific interactive QA application. These Questionnaires were evaluated with four systems, seven analysts, and eight scenarios during a 2-week workshop. Overall, results demonstrate that all three Questionnaires are effective at distinguishing among systems, with the Task Questionnaire being the most sensitive. Results also provide initial support for the validity and reliability of the Questionnaires.
- 1 January 2009
- Published by Cambridge University Press (CUP) in Natural Language Engineering
- Vol. 15 (1) , 119-141
Questionnaires for eliciting evaluation data from users of interactive question answering systems
Evaluating interactive question answering (QA) systems with real users can be challenging because traditional evaluation measures based on the relevance of items returned are difficult to employ since relevance judgments can be unstable in multi-user evaluations. The work reported in this paper evaluates, in distinguishing among a set of interactive QA systems, the effectiveness of three questionnaires: a Cognitive Workload Questionnaire (NASA TLX), and Task and System Questionnaires customized to a specific interactive QA application. These Questionnaires were evaluated with four systems, seven analysts, and eight scenarios during a 2-week workshop. Overall, results demonstrate that all three Questionnaires are effective at distinguishing among systems, with the Task Questionnaire being the most sensitive. Results also provide initial support for the validity and reliability of the Questionnaires.
- 1 December 2006
- Published by Oxford University Press (OUP) in Interacting with Computers
- Vol. 18 (6) , 1186-1214
Evaluation metrics and methodologies for user-centered evaluation of intelligent systems
In the past four years, we have worked with several research programs that were developing intelligent software for use by intelligence analysts. Our involvement in these programs was to develop the metrics and methodologies for assessing the impact on users; in this case, on intelligence analysts. In particular, we focused on metrics to evaluate how much the intelligent systems contribute to the users’ tasks and what the cost is to the user in terms of workload and process deviations. In this paper, we describe the approach used. We started with two types of preliminary investigations – first, collecting and analyzing data from analysts working in an instrumented environment for a period of 2 years, and second, developing and conducting formative evaluations of research software. The long-term studies informed our ideas about the processes that analysts use and provided potential metrics in an environment without intelligent software tools. The formative evaluations helped us to define sets of application-specific metrics. Finally, we conducted assessments during and after technology insertions. We describe the metrics and methodologies used in each of these activities, along with the lessons learned.
- 1 December 2006
- Published by Oxford University Press (OUP) in Interacting with Computers
- Vol. 18 (6) , 1186-1214
Evaluation metrics and methodologies for user-centered evaluation of intelligent systems
In the past four years, we have worked with several research programs that were developing intelligent software for use by intelligence analysts. Our involvement in these programs was to develop the metrics and methodologies for assessing the impact on users; in this case, on intelligence analysts. In particular, we focused on metrics to evaluate how much the intelligent systems contribute to the users’ tasks and what the cost is to the user in terms of workload and process deviations. In this paper, we describe the approach used. We started with two types of preliminary investigations – first, collecting and analyzing data from analysts working in an instrumented environment for a period of 2 years, and second, developing and conducting formative evaluations of research software. The long-term studies informed our ideas about the processes that analysts use and provided potential metrics in an environment without intelligent software tools. The formative evaluations helped us to define sets of application-specific metrics. Finally, we conducted assessments during and after technology insertions. We describe the metrics and methodologies used in each of these activities, along with the lessons learned.
- 1 September 2002
- Published by CNRI Acct in D-Lib Magazine
- Vol. 8 (9)
Evaluation Methodologies for Information Management Systems
- 1 September 2002
- Published by CNRI Acct in D-Lib Magazine
- Vol. 8 (9)
Evaluation Methodologies for Information Management Systems
- 16 November 2001
- Published by Wiley in Journal of the American Society for Information Science and Technology
- Vol. 53 (1) , 28-40
Testing visual information retrieval methodologies case study: Comparative analysis of textual, icon, graphical, and “spring” displays
Although many different visual information retrieval systems have been proposed, few have been tested, and where testing has been performed, results were often inconclusive. Further, there is very little evidence of benchmarking systems against a common standard. An approach for testing novel interfaces is proposed that uses bottom‐up, stepwise testing to allow evaluation of a visualization, itself, rather than restricting evaluation to the system instantiating it. This approach not only makes it easier to control variables, but the tests are also easier to perform. The methodology will be presented through a case study, where a new visualization technique is compared to more traditional ways of presenting data.
- 16 November 2001
- Published by Wiley in Journal of the American Society for Information Science and Technology
- Vol. 53 (1) , 28-40
Testing visual information retrieval methodologies case study: Comparative analysis of textual, icon, graphical, and “spring” displays
Although many different visual information retrieval systems have been proposed, few have been tested, and where testing has been performed, results were often inconclusive. Further, there is very little evidence of benchmarking systems against a common standard. An approach for testing novel interfaces is proposed that uses bottom‐up, stepwise testing to allow evaluation of a visualization, itself, rather than restricting evaluation to the system instantiating it. This approach not only makes it easier to control variables, but the tests are also easier to perform. The methodology will be presented through a case study, where a new visualization technique is compared to more traditional ways of presenting data.