Multiplexed tumor profiling with generative AI accelerates histopathology workflows and improves clinical predictions
Authors:
- Sofia KarkampounaSKSofia Karkampouna
- Francesco BonolloFBFrancesco Bonollo
- Eva CompératECEva Compérat
- Martina RadicMRMartina Radic
- Martin SpahnMSMartin Spahn
- Adriano MartinelliAMAdriano Martinelli
- Martin WartenbergMWMartin Wartenberg
- Marianna Kruithof-De JulioMJMarianna Kruithof-De Julio
- Maria Anna RapsomanikiMRMaria Anna Rapsomaniki
Preprint
Abstract
Understanding the spatial heterogeneity of tumors and its links to disease is a cornerstone of cancer biology. Emerging spatial technologies offer unprecedented capabilities towards this goal, but several limitations hinder their clinical adoption. To date, histopathology workflows still heavily depend on hematoxylin & eosin (H&E) and serial immunohistochemistry (IHC) staining, a cumbersome and tissue-exhaustive process that yields unaligned tissue images. We propose the VirtualMultiplexer, a generative AI toolkit that translates real H&E images to matching IHC images for several markers based on contrastive learning. The VirtualMultiplexer learns from unpaired H&E and IHC images and introduces a novel multi-scale loss to ensure consistent and biologically reliable stainings. The virtually multiplexed images enabled training a Graph Transformer that simultaneously learns from the joint spatial distribution of several markers to predict clinically relevant endpoints. Our results indicate that the VirtualMultiplexer achieves rapid, robust and precise generation of virtually multiplexed imaging datasets of high staining quality that are indistinguishable from the real ones. We successfully employed transfer learning to generate realistic virtual stainings across tissue scales, patient cohorts, and cancer types with no need for model fine-tuning. Crucially, the generated images are not only realistic but also clinically relevant, as they greatly improved the prediction of different clinical endpoints across patient cohorts and cancer types, speeding up histopathology workflows and accelerating spatial biology.Keywords
This publication has 51 references indexed in Scilit:
- Ultra-High-Resolution Unpaired Stain Transformation via Kernelized Instance Normalization2022
- Semi-supervised PR Virtual Staining for Breast Histopathological Images2022
- Region-Guided CycleGANs for Stain Transfer in Whole Slide Images2022
- Pros and cons of GAN evaluation measures: New developmentsComputer Vision and Image Understanding, 2022
- Contrastive Learning for Unpaired Image-to-Image Translation2020
- Going deeper through the Gleason scoring scale: An automatic end-to-end system for histology prostate grading and cribriform pattern detectionComputer Methods and Programs in Biomedicine, 2020
- Use of Deep Learning to Develop and Analyze Computational Hematoxylin and Eosin Staining of Prostate Core Biopsy Images for Tumor DiagnosisJAMA Network Open, 2020
- Deep Profiling of Mouse Splenic Architecture with CODEX Multiplexed ImagingCell, 2018
- Lean-Management im PathologielaborDer Pathologe, 2017
- Skin Tumours2017