Integrating Generative AI into Information Systems Research: A Framework for Synthetic Data Evaluation
Open Access
- 1 January 2025
- conference paper
- Published by HICSS Conference Office
Abstract
Generative AI is paving its way into the research process. Among the plethora of available generative AI solutions, the generation of synthetic data is one of the most controversial. The current division of opinion and the lack of formal approach to AI use in research create a situation of conflicting bad practices and under-used potential. This work aims to add nuance and structure to this research practice by providing a general framework to evaluate the use of synthetic data in different stages of the research process, based on the objective and methods of generation. Relying on a breakout literature review, we explore the fields of Data quality management and Control theory to transfer method theories from these fields to help us build the framework. The resulting conceptual framework provides an iterative scheme where, based on the desired properties of the data and its comparison to the synthetic result, the researcher can improve the outcome of the generation process and, equivalently, formally present the properties that make this data suitable for research.Keywords
This publication has 23 references indexed in Scilit:
- An Evaluation Framework for Synthetic Data Generation Models2024
- Can I trust my fake data – A comprehensive quality assessment framework for synthetic tabular data in healthcareInternational Journal of Medical Informatics, 2024
- The perpetual motion machine of AI-generated data and the distraction of ChatGPT as a ‘scientist’Nature Biotechnology, 2024
- Augmenting research methods with foundation models and generative AIInternational Journal of Information Management, 2024
- Employing large language models in survey researchNatural Language Processing Journal, 2023
- Synthetic data in machine learning for medicine and healthcareNature Biomedical Engineering, 2021
- Designing conceptual articles: four approachesAMS Review, 2020
- The Practitioner's Guide to Data Quality Improvement2011
- A sampling-based computational strategy for the representation of epistemic uncertainty in model predictions with evidence theoryComputer Methods in Applied Mechanics and Engineering, 2007
- Sociology and Social Theory in Agent Based Social Simulation: A SymposiumComputational and Mathematical Organization Theory, 2001