Retrospective validation of MetaSystems’ deep-learning-based digital microscopy platform with assistance compared to manual fluorescence microscopy for detection of mycobacteria
Open Access
- 13 March 2024
- journal article
- research article
- Published by American Society for Microbiology in Journal of Clinical Microbiology
- Vol. 62 (3) , e0106923
Abstract
This study aimed to validate Metasystems’ automated acid-fast bacilli (AFB) smear microscopy scanning and deep-learning-based image analysis module (Neon Metafer) with assistance on respiratory and pleural samples, compared to conventional manual fluorescence microscopy (MM). Analytical parameters were assessed first, followed by a retrospective validation study. In all, 320 archived auramine-O-stained slides selected non-consecutively [85 originally reported as AFB-smear-positive, 235 AFB-smear-negative slides; with an overall mycobacterial culture positivity rate of 24.1% (77/320)] underwent whole-slide imaging and were analyzed by the Metafer Neon AFB Module (version 4.3.130) using a predetermined probability threshold (PT) for AFB detection of 96%. Digital slides were then examined by a trained reviewer blinded to previous AFB smear and culture results, for the final interpretation of assisted digital microscopy (a-DM). Paired results from both microscopic methods were compared to mycobacterial culture. A scanning failure rate of 10.6% (34/320) was observed, leaving 286 slides for analysis. After discrepant analysis, concordance, positive and negative agreements were 95.5% (95%CI, 92.4%–97.6%), 96.2% (95%CI, 89.2%–99.2%), and 95.2% (95%CI, 91.3%–97.7%), respectively. Using mycobacterial culture as reference standard, a-DM and MM had comparable sensitivities: 90.7% (95%CI, 81.7%–96.2%) versus 92.0% (95%CI, 83.4%–97.0%) ( P -value = 1.00); while their specificities differed 91.9% (95%CI, 87.4%–95.2%) versus 95.7% (95%CI, 92.1%–98.0%), respectively ( P -value = 0.03). Using a PT of 96%, MetaSystems’ platform shows acceptable performance. With a national laboratory staff shortage and a local low mycobacterial infection rate, this instrument when combined with culture, can reliably triage-negative AFB-smear respiratory slides and identify positive slides requiring manual confirmation and semi-quantification. IMPORTANCE: This manuscript presents a full validation of MetaSystems’ automated acid-fast bacilli (AFB) smear microscopy scanning and deep-learning-based image analysis module using a probability threshold of 96% including accuracy, precision studies, and evaluation of limit of AFB detection on respiratory samples when the technology is used with assistance. This study is complementary to the conversation started by Tomasello et al. on the use of image analysis artificial intelligence software in routine mycobacterial diagnostic activities within the context of high-throughput laboratories with low incidence of tuberculosis.Keywords
This publication has 30 references indexed in Scilit:
- STARD 2015 guidelines for reporting diagnostic accuracy studies: explanation and elaborationBMJ Open, 2016
- International Clinical Guidelines for the Adoption of Digital Pathology: A Review of Technical AspectsPathobiology, 2016
- STARD 2015: An Updated List of Essential Items for Reporting Diagnostic Accuracy StudiesClinical Chemistry, 2015
- A Review of Automatic Methods Based on Image Processing Techniques for Tuberculosis Detection from Microscopic Sputum Smear ImagesJournal of Medical Systems, 2015
- Validation of Multiple Whole Slide Imaging Scanners Based on the Guideline From the College of American Pathologists Pathology and Laboratory Quality CenterArchives of Pathology & Laboratory Medicine, 2015
- Measuring of Mycobacterium tuberculosis growth: a correlation of the optical measurements with colony forming unitsBrazilian Journal of Microbiology, 2013
- “Proof-Of-Concept” Evaluation of an Automated Sputum Smear Microscopy System for Tuberculosis DiagnosisPLOS ONE, 2012
- Fading of Auramine-Stained Mycobacterial Smears and Implications for External Quality AssuranceJournal of Clinical Microbiology, 2011
- Appropriate statistical methods are required to assess diagnostic tests for replacement, add-on, and triageJournal of Clinical Epidemiology, 2010
- McNemar χ2test revisited: comparing sensitivity and specificity of diagnostic examinationsScandinavian Journal of Clinical and Laboratory Investigation, 2008