Generalising uncertainty improves accuracy and safety of deep learning analytics applied to oncology
Preprint
Abstract
Trust and transparency are critical for deploying deep learning (DL) models into the clinic. DL application poses generalisation obstacles since training/development datasets often have different data distributions to clinical/production datasets that can lead to incorrect predictions with underestimated uncertainty. To investigate this pitfall, we benchmarked one pointwise and three approximate Bayesian DL models used to predict cancer of unknown primary with three independent RNA-seq datasets covering 10,968 samples across 57 primary cancer types. Our results highlight simple and scalable Bayesian DL significantly improves the generalisation of uncertainty estimation (e.g., p-value = 0.0013 for calibration). Moreover, we demonstrate Bayesian DL substantially improves accuracy under data distributional shifts when utilising ‘uncertainty thresholding’ by designing a prototypical metric that evaluates the expected (accuracy) loss when deploying models from development to production, which we call the Area between Development and Production curve (ADP). In summary, Bayesian DL is a hopeful avenue of research for generalising uncertainty, which improves performance, transparency, and therefore safety of DL models for deployment in real-world.Keywords
All Related Versions
- Published version: Scientific Reports, 13 (1), 1.
This publication has 63 references indexed in Scilit:
- Interpretable AI in Healthcare: Enhancing Fairness, Safety, and Trust2022
- Multiomic profiling of checkpoint inhibitor-treated melanoma: Identifying predictors of response and resistance, and markers of biological discordanceCancer Cell, 2022
- Resolving the immune landscape of human prostate at a single-cell level in health and cancerCell Reports, 2021
- A review of uncertainty quantification in deep learning: Techniques, applications and challengesInformation Fusion, 2021
- Critical assessment of methods of protein structure prediction (CASP)—Round XIVProteins: Structure, Function, and Bioinformatics, 2021
- Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methodsMachine Learning, 2021
- CUP-AI-Dx: A tool for inferring cancer tissue of origin and molecular subtype using RNA gene-expression data and artificial intelligenceEBioMedicine, 2020
- Application of a Neural Network Whole Transcriptome–Based Pan-Cancer Method for Diagnosis of Primary and Metastatic CancersJAMA Network Open, 2019
- Cell-of-Origin Patterns Dominate the Molecular Classification of 10,000 Tumors from 33 Types of CancerCell, 2018
- Deep Learning and Its Applications in BiomedicineGenomics, Proteomics & Bioinformatics, 2018