Using Small Language Models to Reverse-Engineer Machine Learning Pipelines Structures
Preprint
Abstract
Background: Extracting the stages that structure Machine Learning (ML) pipelines from source code is key for gaining a deeper understanding of data science practices. However, the diversity caused by the constant evolution of the ML ecosystem (e.g., algorithms, libraries, datasets) makes this task challenging. Existing approaches either depend on non-scalable, manual labeling, or on ML classifiers that do not properly support the diversity of the domain. These limitations highlight the need for more flexible and reliable solutions. Objective: We evaluate whether Small Language Models (SLMs) can leverage their code understanding and classification abilities to address these limitations, and subsequently how they can advance our understanding of data science practices. Method: We conduct a confirmatory study based on two reference works selected for their relevance regarding current state-of-the-art's limitations. First, we compare several SLMs using Cochran's Q test. The best-performing model is then evaluated against the reference studies using two distinct McNemar's tests. We further analyze how variations in taxonomy definitions affect performance through an additional Cochran's Q test. Finally, a goodness-of-fit analysis is conducted using Pearson's chi-squared tests to compare our insights on data science practices with those from prior studies.Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- Towards Evaluation Guidelines for Empirical Studies Involving LLMs2025
- Large Language Models for Software Engineering: A Systematic Literature ReviewACM Transactions on Software Engineering and Methodology, 2024
- Static analysis driven enhancements for comprehension in machine learning notebooksEmpirical Software Engineering, 2024
- Generative AI in Software Engineering Must Be Human-Centered: The Copenhagen ManifestoJournal of Systems and Software, 2024
- Assisted design of data science pipelinesThe VLDB Journal, 2024
- Workflow analysis of data science code in public GitHub repositoriesEmpirical Software Engineering, 2022
- CORAL: COde RepresentAtion learning with weakly-supervised transformers for analyzing data analysisEPJ Data Science, 2022
- Documentation Matters: Human-Centered AI System to Assist Data Science Code Documentation in Computational NotebooksACM Transactions on Computer-Human Interaction, 2022
- CRISP-DM Twenty Years Later: From Data Mining Processes to Data Science TrajectoriesIEEE Transactions on Knowledge and Data Engineering, 2019
- A Survey of Data-Intensive Scientific Workflow ManagementJournal of Grid Computing, 2015