Addressing Threats to Validity in Supervised Machine Learning: A Framework and Best Practices for Education Researchers
Author:
Open Access
- 1 January 2024
- journal article
- research article
- Published by SAGE Publications in AERA Open
- Vol. 10
Abstract
Given the rapid adoption of machine learning methods by education researchers, and the growing acknowledgment of their inherent risks, there is an urgent need for tailored methodological guidance on how to improve and evaluate the validity of inferences drawn from these methods. Drawing on an integrative literature review and extending a well-known framework for theorizing validity in the social sciences, this article provides both an overview of threats to validity in supervised machine learning and plausible approaches for addressing such threats. It collates a list of current best practices, brings supervised learning challenges into a unified conceptual framework, and offers a straightforward reference guide on crucial validity considerations. Finally, it proposes a novel research protocol for researchers to use during project planning and for reviewers and scholars to use when evaluating the validity of supervised machine learning applications.Keywords
This publication has 99 references indexed in Scilit:
- Leakage and the reproducibility crisis in machine-learning-based sciencePatterns, 2023
- Algorithmic Bias in EducationInternational Journal of Artificial Intelligence in Education, 2022
- Overcoming the pitfalls and perils of algorithms: A classification of machine learning biases and mitigation methodsJournal of Business Research, 2022
- How We Code2021
- Large-scale machine learning systems in real-world industrial settings: A review of challenges and solutionsInformation and Software Technology, 2020
- Cross-validation failure: Small sample sizes lead to large error barsNeuroImage, 2018
- When is the Story in the Subgroups?Prevention Science, 2011
- Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within‐study comparisonsJournal of Policy Analysis and Management, 2008
- Learning with Drift Detection2004
- Validity typologies and the logic and practice of quasi‐experimentationNew Directions for Program Evaluation, 1986