Availability and transparency of artificial intelligence models in radiology: a meta-research study

Authors:
Open Access
Publisher website
Google Scholar
Abstract
Objectives: This meta-research study explored the availability of artificial intelligence (AI) models from development studies published in leading radiology journals in 2022, with availability defined as the transparent reporting of relevant technical details, such as model architecture and weights, necessary for independent replication. Materials and methods: A systematic search of Ovid Medline and Embase was conducted to identify AI model development studies published in five leading radiology journals in 2022. Data were extracted on study characteristics, model details, and code and model-sharing practices. The proportion of AI studies sharing their models was analyzed. Logistic regression analyses were employed to explore associations between study characteristics and model availability. Results: Of 268 studies reviewed, 39.9% (n = 107) made their models available. Deep learning (DL) models exhibited particularly low availability, with only 11.5% (n = 13) of the 113 studies being fully available. Training codes for DL models were provided in 22.1% (n = 25), suggesting limited ability to train DL models with one’s own data. Multivariable logistic regression analysis showed that the use of traditional regression-based models (odds ratio [OR], 17.11; 95% CI: 5.52, 53.05; p < 0.001) was associated with higher availability, while the radiomics package usage (OR, 0.27; 95% CI: 0.11, 0.65; p = 0.003) was associated with lower availability. Conclusion: The availability of AI models in radiology publications remains suboptimal, especially for DL models. Enforcing model-sharing policies, enhancing external validation platforms, addressing commercial restrictions, and providing demos for commercial models in open repositories are necessary to improve transparency and replicability in radiology AI research. Key Points: Question The study addresses the limited availability of AI models in radiology, especially DL models, which impacts external validation and clinical reliability. Findings Only 39.9% of radiology AI studies made their models available, with DL models showing particularly low availability at 11.5%. Clinical relevance Improving the availability of radiology AI models is essential for enabling external validation, ensuring reliable clinical application, and advancing patient care by fostering robust and transparent AI systems. Graphical
Funding Information
  • Ministry of Education, Science and Technology (RS-2023-00207978)

This publication has 16 references indexed in Scilit: