Deep Learning Model Reuse in the HuggingFace Community: Challenges, Benefit and Trends
Authors:
- 12 March 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 512-523
Abstract
The ubiquity of large-scale Pre-Trained Models (PTMs) is on the rise, sparking interest in model hubs, and dedicated platforms for hosting PTMs. Despite this trend, a comprehensive exploration of the challenges that users encounter and how the community leverages PTMs remains lacking. To address this gap, we conducted an extensive mixed-methods empirical study by focusing on discussion forums and the model hub of HuggingFace, the largest public model hub. Based on our qualitative analysis, we present a taxonomy of the challenges and benefits associated with PTM reuse within this community. We then conduct a quantitative study to track model-type trends and model documentation evolution over time. Our findings highlight prevalent challenges such as limited guidance for beginner users, struggles with model output comprehensibility in training or inference, and a lack of model understanding. We also identified interesting trends among models where some models maintain high upload rates despite a decline in topics related to them. Additionally, we found that despite the introduction of model documentation tools, its quantity has not increased over time, leading to difficulties in model comprehension and selection among users. Our study sheds light on new challenges in reusing PTMs that were not reported before and we provide recommendations for various stakeholders involved in PTM reuse.Keywords
All Related Versions
Funding Information
- Canadian Institute for Advanced Research (CIFAR) (CRDPJ 537462-18)
- Natural Sciences and Engineering Research Council of Canada (NSERC)
This publication has 21 references indexed in Scilit:
- Exploring the Carbon Footprint of Hugging Face's ML Models: A Repository Mining Study2023
- Reusing Deep Learning Models: Challenges and Directions in Software Engineering2023
- Autonomy Is An Acquired Taste: Exploring Developer Preferences for GitHub Bots2023
- An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry2023
- A mixed-methods analysis of micro-collaborative coding practices in OpenStackEmpirical Software Engineering, 2022
- Onboarding vs. Diversity, Productivity and Quality — Empirical Study of the OpenStack Ecosystem2021
- Pre-trained models: Past, present and futureAI Open, 2021
- An Empirical Study of Common Challenges in Developing Deep Learning Applications2019
- Learning and Transferring Mid-level Image Representations Using Convolutional Neural Networks2014
- What are developers talking about? An analysis of topics and trends in Stack OverflowEmpirical Software Engineering, 2012