A Data-Driven Analysis of Behaviors in Data Curation Processes
Open Access
- 7 February 2023
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Information Systems
- Vol. 41 (3) , 1-35
Abstract
Understanding how data workers interact with data, and various pieces of information related to data preparation, is key to designing systems that can better support them in exploring datasets. To date, however, there is a paucity of research studying the strategies adopted by data workers as they carry out data preparation activities. In this work, we investigate a specific data preparation activity, namely data quality discovery , and aim to (i) understand the behaviors of data workers in discovering data quality issues, (ii) explore what factors (e.g., prior experience) can affect their behaviors, as well as (iii) understand how these behavioral observations relate to their performance. To this end, we collect a multi-modal dataset through a data-driven experiment that relies on the use of eye-tracking technology with a purpose-designed platform built on top of iPython Notebook. The experiment results reveal that: (i) ‘copy–paste–modify’ is a typical strategy for writing code to complete tasks; (ii) proficiency in writing code has a significant impact on the quality of task performance, while perceived difficulty and efficacy can influence task completion patterns; and (iii) searching in external resources is a prevalent action that can be leveraged to achieve better performance. Furthermore, our experiment indicates that providing sample code within the system can help data workers get started with their task, and surfacing underlying data is an effective way to support exploration. By investigating data worker behaviors prior to each search action, we also find that the most common reasons that trigger external search actions are the need to seek assistance in writing or debugging code and to search for relevant code to reuse. Based on our experiment results, we showcase a systematic approach to select from the top best code snippets created by data workers and assemble them to achieve better performance than the best individual performer in the dataset. By doing so, our findings not only provide insights into patterns of interactions with various system components and information resources when performing data curation tasks, but also build effective and efficient data curation processes through data workers’ collective intelligence.Keywords
Funding Information
- ARC Discovery Project (DP190102141)
- ARC Training Centre for Information Resilience (IC200100022)
This publication has 31 references indexed in Scilit:
- How developers search for code: a case study2015
- Clone-based and interactive recommendation for modifying pasted code2015
- Eye‐tracking analysis of user behavior and performance in web search on large and small screensJournal of the Association for Information Science and Technology, 2014
- Measuring and modeling programming experienceEmpirical Software Engineering, 2013
- Task-specific information retrieval systems for software engineersJournal of Computer and System Sciences, 2012
- NASA TLX: Software for assessing subjective mental workloadBehavior Research Methods, 2009
- Nasa-Task Load Index (NASA-TLX); 20 Years LaterProceedings of the Human Factors and Ergonomics Society Annual Meeting, 2006
- Cognitive load theory: implications of cognitive load theory on the design of learningLearning and Instruction, 2002
- Working MemoryScience, 1992
- On a Test of Whether one of Two Random Variables is Stochastically Larger than the OtherThe Annals of Mathematical Statistics, 1947