A practical guide to the implementation of AI in orthopaedic research, Part 6: How to evaluate the performance of AI research?
Open Access
- 31 May 2024
- journal article
- review article
- Published by Wiley in Journal of Experimental Orthopaedics
- Vol. 11 (3) , e12039
Abstract
Artificial intelligence's (AI) accelerating progress demands rigorous evaluation standards to ensure safe, effective integration into healthcare's high‐stakes decisions. As AI increasingly enables prediction, analysis and judgement capabilities relevant to medicine, proper evaluation and interpretation are indispensable. Erroneous AI could endanger patients; thus, developing, validating and deploying medical AI demands adhering to strict, transparent standards centred on safety, ethics and responsible oversight. Core considerations include assessing performance on diverse real‐world data, collaborating with domain experts, confirming model reliability and limitations, and advancing interpretability. Thoughtful selection of evaluation metrics suited to the clinical context along with testing on diverse data sets representing different populations improves generalisability. Partnering software engineers, data scientists and medical practitioners ground assessment in real needs. Journals must uphold reporting standards matching AI's societal impacts. With rigorous, holistic evaluation frameworks, AI can progress towards expanding healthcare access and quality.Level of Evidence: Level V.Keywords
This publication has 52 references indexed in Scilit:
- Shedding Light on the Black Box: Explaining Deep Neural Network Prediction of Clinical OutcomesJournal of Medical Systems, 2021
- Study of ARIMA and least square support vector machine (LS-SVM) models for the prediction of SARS-CoV-2 confirmed cases in the most affected countriesChaos, Solitons & Fractals, 2020
- Improved protein structure prediction using potentials from deep learningNature, 2020
- Clinical applications of machine learning algorithms: beyond the black boxBMJ, 2019
- Dermatologist-level classification of skin cancer with deep neural networksNature, 2017
- Metrics for evaluating 3D medical image segmentation: analysis, selection, and toolBMC Medical Imaging, 2015
- Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD): The TRIPOD StatementEuropean Urology, 2015
- Interrater agreement and interrater reliability: Key concepts, approaches, and applicationsResearch in Social and Administrative Pharmacy, 2013
- Increased vascular penetration and nerve growth in the meniscus: a potential source of pain in osteoarthritisAnnals of the Rheumatic Diseases, 2011
- The local inflammatory environment and microorganisms in “aseptic” loosening of hip prosthesesJournal of Biomedical Materials Research Part B: Applied Biomaterials, 2007