Closing the Knowledge Gap in Designing Data Annotation Interfaces for AI-powered Disaster Management Analytic Systems
Open Access
- 18 March 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 405-418
Abstract
Data annotation interfaces predominantly leverage ground truth labels to guide annotators toward accurate responses. With the growing adoption of Artificial Intelligence (AI) in domain-specific professional tasks, it has become increasingly important to help beginning annotators identify how their early-stage knowledge can lead to inaccurate answers, which in turn, helps to ensure quality annotations at scale. To investigate this issue, we conducted a formative study involving eight individuals from the field of disaster management, each possessing varying levels of expertise. The goal was to understand the prevalent factors contributing to disagreements among annotators when classifying Twitter messages related to disasters and to analyze their respective responses. Our analysis identified two primary causes of disagreement between expert and beginner annotators: 1) a lack of contextual knowledge or uncertainty about the situation, and 2) the absence of visual or supplementary cues. Based on these findings, we designed a Context interface, which generates aids that help beginners identify potential mistakes and provide the hidden context of the presented tweet. The summative study compares Context design with two widely used designs in data annotation UI, Highlight and Reasoning-based interfaces. We found significant differences between these designs in terms of attitudinal and behavioral data. We conclude with implications for designing future interfaces aiming at closing the knowledge gap among annotators.Keywords
All Related Versions
This publication has 39 references indexed in Scilit:
- Modeling and mitigating human annotation errors to design efficient stream processing systems with human-in-the-loop machine learningInternational Journal of Human-Computer Studies, 2022
- Traffic Flow Prediction using Long Short-Term Memory Network and Optimized Spatial Temporal Dependencies2021
- GNES: Learning to Explain Graph Neural Networks2021
- The value of human data annotation for machine learning based anomaly detection in environmental systemsWater Research, 2021
- Building damage annotation on post-hurricane satellite imagery based on convolutional neural networksNatural Hazards, 2020
- An expert–novice comparison of feature choiceApplied Cognitive Psychology, 2020
- Deep Reinforcement Active Learning for Human-in-the-Loop Person Re-Identification2019
- Effect of errors in ground truth on classification accuracyInternational Journal of Remote Sensing, 2009
- Determinants of individual engagement in knowledge sharingThe International Journal of Human Resource Management, 2006
- Direct annotation: a drag-and-drop strategy for labeling photos2002