Nicholas Heller
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Nicholas Heller has a total of 212 co-authors. Here are the top co-authors by number of shared publications:
- Christopher J. Weight
Department of Urology, Integrated Surgical Institute, Cleveland Clinic, Cleveland, OH 44195, USA
11 shared publications Nikolaos Papanikolopoulos
Department of Urology, University of Minnesota, Minneapolis, MN, United States
9 shared publications- Resha Tejpaul
Department of Computer Science and Engineering, University of Minnesota, Minneapolis, Minnesota, USA
6 shared publications - Jack Rickman
Department of Urology, University of Minnesota, Minneapolis, MN, United States
5 shared publications Joshua Dean
Department of Urology, University of Minnesota, Minneapolis, MN, United States
5 shared publicationsBethany Stai
Department of Urology, University of Minnesota, Minneapolis, MN, United States
4 shared publications
Latest Publications
- Open Access
An Automated CT-Derived Marker of Renal Tumor Complexity: The CLARITY Score
Background and Objective: Surgical complexity for renal tumors has traditionally been assessed using manual nephrometry scores, which require unreimbursed physician effort and are subject to interobserver variability. This study introduces an objective, fully automated alternative derived from decades of experience at a large academic center. Methods: We trained a CT classification model to predict whether a patient would ultimately undergo Partial or Radical Nephrectomy (PN or RN). We hypothesized that the model’s confidence in RN (termed the CLARITY score) would serve as a surrogate for the difficulty of nephron-sparing approaches and thus for tumor complexity. This hypothesis was tested using multivariate logistic regression for failure to achieve trifecta, estimated blood loss (EBL) ≥ 500 mL, and length of stay ≥ 3 d. CLARITY was compared with tumor size and R.E.N.A.L. score. External validation in a geographically distinct cohort was performed. Key Findings and Limitations: For predicting RN, CLARITY achieved an AUROC of 0.899 internally and 0.898 externally. In the external PN subgroup, it outperformed tumor size and R.E.N.A.L. score in predicting failure to achieve trifecta (AUROC 0.613), EBL ≥ 500 mL (0.727), and length of stay ≥ 3 d (0.673). In multivariable analysis, CLARITY remained associated with each outcome, whereas R.E.N.A.L. and size were not. This study is limited by its retrospective design. Conclusions and Clinical Implications: CLARITY is an automated CT-derived marker that quantifies renal tumor complexity more effectively than tumor size and R.E.N.A.L. score and may support scalable, objective preoperative complexity assessment. To support reproducibility and external validation, we have released a public inference pipeline and web-based DICOM upload portal for research use.
Open AccessAn Automated CT-Derived Marker of Renal Tumor Complexity: The CLARITY Score
Background and Objective: Surgical complexity for renal tumors has traditionally been assessed using manual nephrometry scores, which require unreimbursed physician effort and are subject to interobserver variability. This study introduces an objective, fully automated alternative derived from decades of experience at a large academic center. Methods: We trained a CT classification model to predict whether a patient would ultimately undergo Partial or Radical Nephrectomy (PN or RN). We hypothesized that the model’s confidence in RN (termed the CLARITY score) would serve as a surrogate for the difficulty of nephron-sparing approaches and thus for tumor complexity. This hypothesis was tested using multivariate logistic regression for failure to achieve trifecta, estimated blood loss (EBL) ≥ 500 mL, and length of stay ≥ 3 d. CLARITY was compared with tumor size and R.E.N.A.L. score. External validation in a geographically distinct cohort was performed. Key Findings and Limitations: For predicting RN, CLARITY achieved an AUROC of 0.899 internally and 0.898 externally. In the external PN subgroup, it outperformed tumor size and R.E.N.A.L. score in predicting failure to achieve trifecta (AUROC 0.613), EBL ≥ 500 mL (0.727), and length of stay ≥ 3 d (0.673). In multivariable analysis, CLARITY remained associated with each outcome, whereas R.E.N.A.L. and size were not. This study is limited by its retrospective design. Conclusions and Clinical Implications: CLARITY is an automated CT-derived marker that quantifies renal tumor complexity more effectively than tumor size and R.E.N.A.L. score and may support scalable, objective preoperative complexity assessment. To support reproducibility and external validation, we have released a public inference pipeline and web-based DICOM upload portal for research use.
- Open Access
- 2 April 2026
- Published by Wiley in BJU International
- Vol. 138 (2) , 244-249
Reducing complexity in International Bladder Cancer Group intermediate‐risk non‐muscle‐invasive bladder cancer stratification: a three‐factor approach
Objectives: To evaluate whether a simplified risk model preserves the prognostic performance of the International Bladder Cancer Group (IBCG) five‐factor risk stratification model in intermediate‐risk (IR) non‐muscle‐invasive bladder cancer (NMIBC). Patients and Methods: We performed a multicentre retrospective analysis of 2822 patients with IR NMIBC treated with transurethral resection of bladder tumour and intravesical therapy between 2005 and 2025. Multivariable Cox regression identified independent predictors of recurrence and progression. A simplified risk model was constructed using significant variables and compared with the full IBCG model using recurrence and progression rates, area under the curve analysis, and likelihood ratio testing. Results: A total of 2822 patients with IR NMIBC were included. When classified using the full IBCG model, 1143 (41%) were in the IR‐low, 1535 (54%) IR‐intermediate, and 144 (5%) IR‐high‐risk groups. On multivariable analysis, only multifocality, early recurrence, and failure of intravesical therapy independently predicted oncological outcomes. Using these factors, patients were stratified as IR‐low (no factors), IR‐intermediate (one factor), or IR‐high (two or more factors). Under the simplified model, 1535 (54%) patients were classified as IR‐low, 906 (32%) as IR‐intermediate, and 381 (14%) as IR‐high. The 3‐year recurrence and progression rates increased stepwise across simplified risk groups. Discriminative performance of the simplified model was comparable to the full model across all metrics, with no meaningful loss of discrimination. Conclusion: Three variables captured the most prognostic signal in this cohort without loss of discrimination compared with the full model. This simplified model is an internally validated refinement, with planned external validation to confirm generalisability and to ensure that simplified thresholds do not alter the clinically important IR‐high subgroup used in care and trials.
- 2 April 2026
- Published by Wiley in BJU International
- Vol. 138 (2) , 244-249
Open AccessReducing complexity in International Bladder Cancer Group intermediate‐risk non‐muscle‐invasive bladder cancer stratification: a three‐factor approach
Objectives: To evaluate whether a simplified risk model preserves the prognostic performance of the International Bladder Cancer Group (IBCG) five‐factor risk stratification model in intermediate‐risk (IR) non‐muscle‐invasive bladder cancer (NMIBC). Patients and Methods: We performed a multicentre retrospective analysis of 2822 patients with IR NMIBC treated with transurethral resection of bladder tumour and intravesical therapy between 2005 and 2025. Multivariable Cox regression identified independent predictors of recurrence and progression. A simplified risk model was constructed using significant variables and compared with the full IBCG model using recurrence and progression rates, area under the curve analysis, and likelihood ratio testing. Results: A total of 2822 patients with IR NMIBC were included. When classified using the full IBCG model, 1143 (41%) were in the IR‐low, 1535 (54%) IR‐intermediate, and 144 (5%) IR‐high‐risk groups. On multivariable analysis, only multifocality, early recurrence, and failure of intravesical therapy independently predicted oncological outcomes. Using these factors, patients were stratified as IR‐low (no factors), IR‐intermediate (one factor), or IR‐high (two or more factors). Under the simplified model, 1535 (54%) patients were classified as IR‐low, 906 (32%) as IR‐intermediate, and 381 (14%) as IR‐high. The 3‐year recurrence and progression rates increased stepwise across simplified risk groups. Discriminative performance of the simplified model was comparable to the full model across all metrics, with no meaningful loss of discrimination. Conclusion: Three variables captured the most prognostic signal in this cohort without loss of discrimination compared with the full model. This simplified model is an internally validated refinement, with planned external validation to confirm generalisability and to ensure that simplified thresholds do not alter the clinically important IR‐high subgroup used in care and trials.
- Open Access
- 27 February 2026
- Published by Wiley in BJUI Compass
- Vol. 7 (3)
Preoperative predictors of adverse pathology and recurrence‐free survival for patients with renal masses
Objectives: Our objective was to develop algorithms to predict adverse pathology (AP) and recurrence‐free survival (RFS) for patients with renal tumours primarily based on multifaceted analysis of preoperative CT imaging. Patients/Methods: Seven hundred forty‐eight patients with non‐metastatic renal tumours managed with definitive surgery at Cleveland Clinic (2011–2014) were retrospectively evaluated (median follow‐up 9.1 years). All patients underwent contrast‐enhanced CT and parenchymal volume analysis using semi‐automated software. A variety of conventional radiological features were evaluated in addition to parenchymal volume replacement (PVR) due to invasive tumour growth, using the contralateral kidney as a control. Adverse pathology (AP) was defined as stage ≥pT3a, grade 3/4 or sarcomatoid/rhabdoid features. Multivariable logistic regression and Cox proportional hazards regression analyses were used to develop predictive models. Results: Overall, 339/748 patients (45%) had AP, which significantly associated with reduced RFS. On univariable analysis, tumour‐size, degree of vascularity, heterogeneity, irregular contour, sinus margin irregularity, necrosis, non‐cystic tumour and increased PVR significantly associated with AP. On multivariable logistic regression, male sex, R.E.N.A.L. Nearness, heterogeneity, necrosis, sinus margin irregularity and PVR ≥ 25% independently associated with AP. Multivariable analysis indicated that tumour size, heterogeneity, necrosis, PVR ≥ 25% and tumour‐related symptoms significantly associated with reduced RFS. Models for AP and RFS at 3, 5 and 10 years showed area under the curve (AUC) values of 0.81 and 0.84–0.86, respectively. Conclusions: These findings confirm that radiological features and PVR are associated with AP and reduced RFS after definitive renal cancer surgery. Our predictive models are entirely based on preoperative parameters and may improve patient counselling and occasionally preclude the need for renal mass biopsy.
- 27 February 2026
- Published by Wiley in BJUI Compass
- Vol. 7 (3)
Open AccessPreoperative predictors of adverse pathology and recurrence‐free survival for patients with renal masses
Objectives: Our objective was to develop algorithms to predict adverse pathology (AP) and recurrence‐free survival (RFS) for patients with renal tumours primarily based on multifaceted analysis of preoperative CT imaging. Patients/Methods: Seven hundred forty‐eight patients with non‐metastatic renal tumours managed with definitive surgery at Cleveland Clinic (2011–2014) were retrospectively evaluated (median follow‐up 9.1 years). All patients underwent contrast‐enhanced CT and parenchymal volume analysis using semi‐automated software. A variety of conventional radiological features were evaluated in addition to parenchymal volume replacement (PVR) due to invasive tumour growth, using the contralateral kidney as a control. Adverse pathology (AP) was defined as stage ≥pT3a, grade 3/4 or sarcomatoid/rhabdoid features. Multivariable logistic regression and Cox proportional hazards regression analyses were used to develop predictive models. Results: Overall, 339/748 patients (45%) had AP, which significantly associated with reduced RFS. On univariable analysis, tumour‐size, degree of vascularity, heterogeneity, irregular contour, sinus margin irregularity, necrosis, non‐cystic tumour and increased PVR significantly associated with AP. On multivariable logistic regression, male sex, R.E.N.A.L. Nearness, heterogeneity, necrosis, sinus margin irregularity and PVR ≥ 25% independently associated with AP. Multivariable analysis indicated that tumour size, heterogeneity, necrosis, PVR ≥ 25% and tumour‐related symptoms significantly associated with reduced RFS. Models for AP and RFS at 3, 5 and 10 years showed area under the curve (AUC) values of 0.81 and 0.84–0.86, respectively. Conclusions: These findings confirm that radiological features and PVR are associated with AP and reduced RFS after definitive renal cancer surgery. Our predictive models are entirely based on preoperative parameters and may improve patient counselling and occasionally preclude the need for renal mass biopsy.
- 3 July 2025
- Published by Wiley in BJU International
Degree of contrast enhancement improves prediction of new baseline glomerular filtration rate following radical nephrectomy
Objectives: To assess the accuracy of PVA+, a model integrating differential degree of enhancement (DOE) with parenchymal volume analysis (PVA), for predicting new baseline glomerular filtration rate (NBGFR) following radical nephrectomy (RN), and to assess PVA+ vs PVA alone in patients with hydronephrosis, where disruptions in the parenchymal volume/function relationship may impact prediction accuracy.Patients and Methods: A total of 485 RN patients (2006–2021) with preoperative contrast‐enhanced computed tomography were included. The predicted NBGFR was estimated to be 1.25 (GFRpreRN x split renal function [SRF]contralateral), with 1.25 representing the average renal functional compensation following RN in adults. For PVA, SRF was determined with differential parenchymal volumes, whereas for PVA+ the differential DOE was also incorporated. Parenchymal volumes and DOE were measured using semi‐automated software from the venous phase vs non‐contrast images. The accuracy of predicted vs observed NBGFR was compared using a 15% threshold.Results: Overall, PVA+ accurately predicted NBGFR for 76% (r = 0.86) of the cases, compared to 72% for PVA alone (r = 0.84, both P < 0.05). PVA+ was particularly advantageous in patients with hydronephrosis or renal vein thrombosis (RVT; 68% vs 50% accuracy for PVA+ vs PVA alone; P < 0.01). Similarly, PVA+ was superior in patients with infiltrative renal masses (IRMs) compared to PVA alone (55% vs 40%; P < 0.01).Conclusion: Accurate prediction of NBGFR following RN is essential for counselling patients for whom RN is being considered vs partial nephrectomy. PVA+ was superior to PVA alone for NBGFR prediction, particularly in patients with hydronephrosis, RVT and IRMs. Requiring only routine contrast‐enhanced preoperative imaging, PVA+ improves peri‐operative decision‐making for patients with localised renal masses, particularly in cases where the parenchymal volume/function relationship might be distorted.
- 3 July 2025
- Published by Wiley in BJU International
Degree of contrast enhancement improves prediction of new baseline glomerular filtration rate following radical nephrectomy
Objectives: To assess the accuracy of PVA+, a model integrating differential degree of enhancement (DOE) with parenchymal volume analysis (PVA), for predicting new baseline glomerular filtration rate (NBGFR) following radical nephrectomy (RN), and to assess PVA+ vs PVA alone in patients with hydronephrosis, where disruptions in the parenchymal volume/function relationship may impact prediction accuracy.Patients and Methods: A total of 485 RN patients (2006–2021) with preoperative contrast‐enhanced computed tomography were included. The predicted NBGFR was estimated to be 1.25 (GFRpreRN x split renal function [SRF]contralateral), with 1.25 representing the average renal functional compensation following RN in adults. For PVA, SRF was determined with differential parenchymal volumes, whereas for PVA+ the differential DOE was also incorporated. Parenchymal volumes and DOE were measured using semi‐automated software from the venous phase vs non‐contrast images. The accuracy of predicted vs observed NBGFR was compared using a 15% threshold.Results: Overall, PVA+ accurately predicted NBGFR for 76% (r = 0.86) of the cases, compared to 72% for PVA alone (r = 0.84, both P < 0.05). PVA+ was particularly advantageous in patients with hydronephrosis or renal vein thrombosis (RVT; 68% vs 50% accuracy for PVA+ vs PVA alone; P < 0.01). Similarly, PVA+ was superior in patients with infiltrative renal masses (IRMs) compared to PVA alone (55% vs 40%; P < 0.01).Conclusion: Accurate prediction of NBGFR following RN is essential for counselling patients for whom RN is being considered vs partial nephrectomy. PVA+ was superior to PVA alone for NBGFR prediction, particularly in patients with hydronephrosis, RVT and IRMs. Requiring only routine contrast‐enhanced preoperative imaging, PVA+ improves peri‐operative decision‐making for patients with localised renal masses, particularly in cases where the parenchymal volume/function relationship might be distorted.
- Open Access
Lack of children in public medical imaging data points to growing age bias in biomedical AI
Artificial intelligence (AI) is rapidly transforming healthcare, but its benefits are not reaching all patients equally. Children remain overlooked with only 17% of FDA-approved medical AI devices labeled for pediatric use. In this work, we demonstrate that this exclusion may stem from a fundamental data gap. Our systematic review of 181 public medical imaging datasets reveals that children represent just under 1% of available data, while the majority of machine learning imaging conference papers we surveyed utilized publicly available data for methods development. Much like systematic biases of other kinds in model development, past studies have demonstrated the manner in which pediatric representation in data used for models intended for the pediatric population is essential for model performance in that population. We add to these findings, showing that adult-trained chest radiograph models exhibit significant age bias when applied to pediatric populations, with higher false positive rates in younger children. This work underscores the urgent need for increased pediatric representation in publicly accessible medical datasets. We provide actionable recommendations for researchers, policymakers, and data curators to address this age equity gap and ensure AI benefits patients of all ages. 1-2 sentence summary Our analysis reveals a critical healthcare age disparity: children represent less than 1% of public medical imaging datasets. This gap in representation leads to biased predictions across medical image foundation models, with the youngest patients facing the highest risk of misdiagnosis.
Open AccessLack of children in public medical imaging data points to growing age bias in biomedical AI
Artificial intelligence (AI) is rapidly transforming healthcare, but its benefits are not reaching all patients equally. Children remain overlooked with only 17% of FDA-approved medical AI devices labeled for pediatric use. In this work, we demonstrate that this exclusion may stem from a fundamental data gap. Our systematic review of 181 public medical imaging datasets reveals that children represent just under 1% of available data, while the majority of machine learning imaging conference papers we surveyed utilized publicly available data for methods development. Much like systematic biases of other kinds in model development, past studies have demonstrated the manner in which pediatric representation in data used for models intended for the pediatric population is essential for model performance in that population. We add to these findings, showing that adult-trained chest radiograph models exhibit significant age bias when applied to pediatric populations, with higher false positive rates in younger children. This work underscores the urgent need for increased pediatric representation in publicly accessible medical datasets. We provide actionable recommendations for researchers, policymakers, and data curators to address this age equity gap and ensure AI benefits patients of all ages. 1-2 sentence summary Our analysis reveals a critical healthcare age disparity: children represent less than 1% of public medical imaging datasets. This gap in representation leads to biased predictions across medical image foundation models, with the youngest patients facing the highest risk of misdiagnosis.
- Open Access
- 26 May 2025
- Published by MDPI AG in Journal of Imaging
- Vol. 11 (6) , 174
A Study on Energy Consumption in AI-Driven Medical Image Segmentation
As artificial intelligence advances in medical image analysis, its environmental impact remains largely overlooked. This study analyzes the energy demands of AI workflows for medical image segmentation using the popular Kidney Tumor Segmentation-2019 (KiTS-19) dataset. It examines how training and inference differ in energy consumption, focusing on factors that influence resource usage, such as computational complexity, memory access, and I/O operations. To address these aspects, we evaluated three variants of convolution—Standard Convolution, Depthwise Convolution, and Group Convolution—combined with optimization techniques such as Mixed Precision and Gradient Accumulation. While training is energy-intensive, the recurring nature of inference often results in significantly higher cumulative energy consumption over a model’s life cycle. Depthwise Convolution with Mixed Precision achieves the lowest energy consumption during training while maintaining strong performance, making it the most energy-efficient configuration among those tested. In contrast, Group Convolution fails to achieve energy efficiency due to significant input/output overhead. These findings emphasize the need for GPU-centric strategies and energy-conscious AI practices, offering actionable guidance for designing scalable, sustainable innovation in medical image analysis.
- 26 May 2025
- Published by MDPI AG in Journal of Imaging
- Vol. 11 (6) , 174
Open AccessA Study on Energy Consumption in AI-Driven Medical Image Segmentation
As artificial intelligence advances in medical image analysis, its environmental impact remains largely overlooked. This study analyzes the energy demands of AI workflows for medical image segmentation using the popular Kidney Tumor Segmentation-2019 (KiTS-19) dataset. It examines how training and inference differ in energy consumption, focusing on factors that influence resource usage, such as computational complexity, memory access, and I/O operations. To address these aspects, we evaluated three variants of convolution—Standard Convolution, Depthwise Convolution, and Group Convolution—combined with optimization techniques such as Mixed Precision and Gradient Accumulation. While training is energy-intensive, the recurring nature of inference often results in significantly higher cumulative energy consumption over a model’s life cycle. Depthwise Convolution with Mixed Precision achieves the lowest energy consumption during training while maintaining strong performance, making it the most energy-efficient configuration among those tested. In contrast, Group Convolution fails to achieve energy efficiency due to significant input/output overhead. These findings emphasize the need for GPU-centric strategies and energy-conscious AI practices, offering actionable guidance for designing scalable, sustainable innovation in medical image analysis.
- Open Access
- 19 March 2025
- Published by Wiley in BJUI Compass
- Vol. 6 (3) , e70013
Renal parenchymal volume analysis: Clinical and research applications
Background and Objectives: In most patients, the renal parenchymal volumes in each kidney directly correlate with function and can be used as a proxy to determine split renal function (SRF). This simple principle forms the basis for parenchymal volume analysis (PVA) with semiautomated software, which can be leveraged to predict SRF and new‐baseline glomerular filtration rate (NBGFR) following nephrectomy. PVA was originally used to evaluate renal transplantation donors and has replaced nuclear renal scans (NRS) in this domain. PVA has subsequently been explored for the management of patients with kidney cancer for whom difficult decisions about radical versus partial nephrectomy can be influenced by accurate prediction of NBGFR. Our objective is to present a comprehensive review of the applications of PVA in urology including their clinical and research implications.Methods: Key articles utilizing renal PVA to improve clinical care and facilitate urologic research were reviewed with special emphasis on take‐home points of clinical relevance and their contributions to progress in the field.Results: There have been considerable advances in renal PVA over the past 15 years, which is now established as a reference standard for the prediction of functional outcomes after renal surgery. PVA provides improved accuracy when compared to NRS‐based estimates or non‐SRF‐based algorithms. PVA can be performed in minutes using routine preoperative cross‐sectional imaging and can be readily applied at the point of care. Additionally, PVA has important research applications, enabling the precise study of the determinants of functional recovery after partial nephrectomy, which can affect surgical approaches to this procedure.Conclusions: Despite the wide availability of PVA, primarily for use in renal transplantation, it has not been widely implemented for other urologic purposes at most centres. Our hope is that this narrative review will increase PVA utilization in urology and facilitate further progress in the field.
- 19 March 2025
- Published by Wiley in BJUI Compass
- Vol. 6 (3) , e70013
Open AccessRenal parenchymal volume analysis: Clinical and research applications
Background and Objectives: In most patients, the renal parenchymal volumes in each kidney directly correlate with function and can be used as a proxy to determine split renal function (SRF). This simple principle forms the basis for parenchymal volume analysis (PVA) with semiautomated software, which can be leveraged to predict SRF and new‐baseline glomerular filtration rate (NBGFR) following nephrectomy. PVA was originally used to evaluate renal transplantation donors and has replaced nuclear renal scans (NRS) in this domain. PVA has subsequently been explored for the management of patients with kidney cancer for whom difficult decisions about radical versus partial nephrectomy can be influenced by accurate prediction of NBGFR. Our objective is to present a comprehensive review of the applications of PVA in urology including their clinical and research implications.Methods: Key articles utilizing renal PVA to improve clinical care and facilitate urologic research were reviewed with special emphasis on take‐home points of clinical relevance and their contributions to progress in the field.Results: There have been considerable advances in renal PVA over the past 15 years, which is now established as a reference standard for the prediction of functional outcomes after renal surgery. PVA provides improved accuracy when compared to NRS‐based estimates or non‐SRF‐based algorithms. PVA can be performed in minutes using routine preoperative cross‐sectional imaging and can be readily applied at the point of care. Additionally, PVA has important research applications, enabling the precise study of the determinants of functional recovery after partial nephrectomy, which can affect surgical approaches to this procedure.Conclusions: Despite the wide availability of PVA, primarily for use in renal transplantation, it has not been widely implemented for other urologic purposes at most centres. Our hope is that this narrative review will increase PVA utilization in urology and facilitate further progress in the field.
- Open Access
- 25 October 2024
- Published by Springer Nature
- p. 52-69
Quality Assured: Rethinking Annotation Strategies in Imaging AI
This paper does not describe a novel method. Instead, it studies an essential foundation for reliable benchmarking and ultimately real-world application of AI-based image analysis: generating high-quality reference annotations. Previous research has focused on crowdsourcing as a means of outsourcing annotations. However, little attention has so far been given to annotation companies, specifically regarding their internal quality assurance (QA) processes. Therefore, our aim is to evaluate the influence of QA employed by annotation companies on annotation quality and devise methodologies for maximizing data annotation efficacy. Based on a total of 57,648 instance segmented images obtained from a total of 924 annotators and 34 QA workers from four annotation companies and Amazon Mechanical Turk (MTurk), we derived the following insights: (1) Annotation companies perform better both in terms of quantity and quality compared to the widely used platform MTurk. (2) Annotation companies’ internal QA only provides marginal improvements, if any. However, improving labeling instructions instead of investing in QA can substantially boost annotation performance. (3) The benefit of internal QA depends on specific image characteristics. Our work could enable researchers to derive substantially more value from a fixed annotation budget and change the way annotation companies conduct internal QA.
- 25 October 2024
- Published by Springer Nature
- p. 52-69
Open AccessQuality Assured: Rethinking Annotation Strategies in Imaging AI
This paper does not describe a novel method. Instead, it studies an essential foundation for reliable benchmarking and ultimately real-world application of AI-based image analysis: generating high-quality reference annotations. Previous research has focused on crowdsourcing as a means of outsourcing annotations. However, little attention has so far been given to annotation companies, specifically regarding their internal quality assurance (QA) processes. Therefore, our aim is to evaluate the influence of QA employed by annotation companies on annotation quality and devise methodologies for maximizing data annotation efficacy. Based on a total of 57,648 instance segmented images obtained from a total of 924 annotators and 34 QA workers from four annotation companies and Amazon Mechanical Turk (MTurk), we derived the following insights: (1) Annotation companies perform better both in terms of quantity and quality compared to the widely used platform MTurk. (2) Annotation companies’ internal QA only provides marginal improvements, if any. However, improving labeling instructions instead of investing in QA can substantially boost annotation performance. (3) The benefit of internal QA depends on specific image characteristics. Our work could enable researchers to derive substantially more value from a fixed annotation budget and change the way annotation companies conduct internal QA.
- Open Access
Quality Assured: Rethinking Annotation Strategies in Imaging AI
This paper does not describe a novel method. Instead, it studies an essential foundation for reliable benchmarking and ultimately real-world application of AI-based image analysis: generating high-quality reference annotations. Previous research has focused on crowdsourcing as a means of outsourcing annotations. However, little attention has so far been given to annotation companies, specifically regarding their internal quality assurance (QA) processes. Therefore, our aim is to evaluate the influence of QA employed by annotation companies on annotation quality and devise methodologies for maximizing data annotation efficacy. Based on a total of 57,648 instance segmented images obtained from a total of 924 annotators and 34 QA workers from four annotation companies and Amazon Mechanical Turk (MTurk), we derived the following insights: (1) Annotation companies perform better both in terms of quantity and quality compared to the widely used platform MTurk. (2) Annotation companies' internal QA only provides marginal improvements, if any. However, improving labeling instructions instead of investing in QA can substantially boost annotation performance. (3) The benefit of internal QA depends on specific image characteristics. Our work could enable researchers to derive substantially more value from a fixed annotation budget and change the way annotation companies conduct internal QA.
Open AccessQuality Assured: Rethinking Annotation Strategies in Imaging AI
This paper does not describe a novel method. Instead, it studies an essential foundation for reliable benchmarking and ultimately real-world application of AI-based image analysis: generating high-quality reference annotations. Previous research has focused on crowdsourcing as a means of outsourcing annotations. However, little attention has so far been given to annotation companies, specifically regarding their internal quality assurance (QA) processes. Therefore, our aim is to evaluate the influence of QA employed by annotation companies on annotation quality and devise methodologies for maximizing data annotation efficacy. Based on a total of 57,648 instance segmented images obtained from a total of 924 annotators and 34 QA workers from four annotation companies and Amazon Mechanical Turk (MTurk), we derived the following insights: (1) Annotation companies perform better both in terms of quantity and quality compared to the widely used platform MTurk. (2) Annotation companies' internal QA only provides marginal improvements, if any. However, improving labeling instructions instead of investing in QA can substantially boost annotation performance. (3) The benefit of internal QA depends on specific image characteristics. Our work could enable researchers to derive substantially more value from a fixed annotation budget and change the way annotation companies conduct internal QA.
- 14 March 2024
- Published by SAGE Publications in Dermatitis®
- Vol. 35 (2) , 144-148
Demonstration of Convolutional Neural Networks to Determine Patch Test Reactivity
: Background: Convolutional neural networks (CNNs) have the potential to assist allergists and dermatologists in the analysis of patch tests. Such models can help reduce interprovider variability and improve consistency of patch test interpretations. Objective: Our aim is to evaluate the performance of a CNN model as a proof of concept in discriminating between patch tests with reactions and patch tests without reactions. Methods: We performed a retrospective analysis of patch test images from March 2020 to March 2021. The CNN model was trained as a binary classifier to discriminate between reaction and nonreaction patches. Performance of the model was determined using summary statistics and receiver operator characteristics (ROC) curves. Results: In total, 13,622 images from 125 patients were recorded for analysis. The majority of patients in the cohort were female (81.6%) with Fitzpatrick skin types I–II (88.0%). The area under curve was 0.940, indicating a high discriminative performance of the model for this data set. This resulted in a total accuracy of 90.1%, sensitivity of 86.0%, and specificity of 90.2%. Conclusions: CNNs have the capacity to determine the presence of delayed-type reactions in patch tests. Future prospective studies are required to assess the generalizability of such models.
- 14 March 2024
- Published by SAGE Publications in Dermatitis®
- Vol. 35 (2) , 144-148
Demonstration of Convolutional Neural Networks to Determine Patch Test Reactivity
: Background: Convolutional neural networks (CNNs) have the potential to assist allergists and dermatologists in the analysis of patch tests. Such models can help reduce interprovider variability and improve consistency of patch test interpretations. Objective: Our aim is to evaluate the performance of a CNN model as a proof of concept in discriminating between patch tests with reactions and patch tests without reactions. Methods: We performed a retrospective analysis of patch test images from March 2020 to March 2021. The CNN model was trained as a binary classifier to discriminate between reaction and nonreaction patches. Performance of the model was determined using summary statistics and receiver operator characteristics (ROC) curves. Results: In total, 13,622 images from 125 patients were recorded for analysis. The majority of patients in the cohort were female (81.6%) with Fitzpatrick skin types I–II (88.0%). The area under curve was 0.940, indicating a high discriminative performance of the model for this data set. This resulted in a total accuracy of 90.1%, sensitivity of 86.0%, and specificity of 90.2%. Conclusions: CNNs have the capacity to determine the presence of delayed-type reactions in patch tests. Future prospective studies are required to assess the generalizability of such models.