Subjectivity in Unsupervised Machine Learning Model Selection
Authors:
Open Access
- 20 May 2024
- journal article
- research article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Symposium Series
- Vol. 3 (1) , 22-29
Abstract
Model selection is a necessary step in unsupervised machine learning. Despite numerous criteria and metrics, model selection remains subjective. A high degree of subjectivity may lead to questions about repeatability and reproducibility of various machine learning studies and doubts about the robustness of models deployed in the real world. Yet, the impact of modelers' preferences on model selection outcomes remains largely unexplored. This study uses the Hidden Markov Model as an example to investigate the subjectivity involved in model selection. We asked 33 participants and three Large Language Models (LLMs) to make model selections in three scenarios. Results revealed variability and inconsistencies in both the participants’ and the LLMs' choices, especially when different criteria and metrics disagree. Sources of subjectivity include varying opinions on the importance of different criteria and metrics, differing views on how parsimonious a model should be, and how the size of a dataset should influence model selection. The results underscore the importance of developing a more standardized way to document subjective choices made in model selection processes.Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- Subjectivity in the Creation of Machine Learning ModelsJournal of Data and Information Quality, 2021
- The influence of hidden researcher decisions in applied microeconomicsEconomic Inquiry, 2021
- Sensitivity and specificity of information criteriaBriefings in Bioinformatics, 2020
- Selecting the Number of States in Hidden Markov Models: Pragmatic Solutions Illustrated Using Animal MovementJournal of Agricultural, Biological and Environmental Statistics, 2017
- Hidden Markov Model-Based Fault Detection Approach for a Multimode ProcessIndustrial & Engineering Chemistry Research, 2016
- A comparison of some criteria for states selection in the latent Markov model for longitudinal dataAdvances in Data Analysis and Classification, 2013
- False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant2011
- Measuring an Individual's Tendency to Take Risks: The Risk Propensity Scale1Journal of Applied Social Psychology, 2008
- Selecting hidden Markov model state number with cross-validated likelihoodComputational Statistics, 2007
- A new look at the statistical model identificationIEEE Transactions on Automatic Control, 1974