Sayantan Kumar
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Sayantan Kumar has a total of 74 co-authors. Here are the top co-authors by number of shared publications:
- Aristeidis Sotiras
University of Pennsylvania
13 shared publications - Philip Richard Orrin Payne
Institute for Informatics, Data Science and Biostatistics, Washington University in St. Louis School of Medicine , St. Louis, MO 63110
10 shared publications - Braden Yang
Mallinckrodt Institute of Radiology Washington University School of Medicine in St Louis Saint Louis Missouri USA
6 shared publications Deydeep Kothapalli
Mallinckrodt Institute of Radiology Washington University School of Medicine in St Louis Saint Louis Missouri USA
6 shared publications- Peijie Qiu
Mallinckrodt Institute of Radiology, Washington University School of Medicine, St. Louis, MO, USA
6 shared publications - Brian Andrew Gordon
Mallinckrodt Institute of Radiology, Washington University School of Medicine, St. Louis, MO 63110, USA
5 shared publications
Latest Publications
- Open Access
Temporally Phenotyping GLP-1RA Case Reports with Large Language Models: A Textual Time Series Corpus and Risk Modeling
Type 2 diabetes case reports describe complex clinical courses, but their timelines are often expressed in language that is difficult to reuse in longitudinal modeling. To address this gap, we developed a textual time-series corpus of 136 PubMed Open Access single-patient case reports involving glucagon-like peptide 1 receptor agonists, with clinical events associated with their most probable reference times. We evaluated automated LLM timeline extraction against gold-standard timelines annotated by clinical domain experts, assessing how well systems recovered clinical events and their timings. The best-performing LLM produced high event coverage (GPT5; 0.871) and reliable temporal sequencing across symptoms (GPT5; 0.843), diagnoses, treatments, laboratory tests, and outcomes. As a downstream demonstration, time-to-event analyses in diabetes suggested lower risk of respiratory sequelae among GLP-1 users versus non-users (HR=0.259, p!0.05), consistent with prior reports of improved respiratory outcomes. Temporal annotations and code will be released upon acceptance.
Open AccessTemporally Phenotyping GLP-1RA Case Reports with Large Language Models: A Textual Time Series Corpus and Risk Modeling
Type 2 diabetes case reports describe complex clinical courses, but their timelines are often expressed in language that is difficult to reuse in longitudinal modeling. To address this gap, we developed a textual time-series corpus of 136 PubMed Open Access single-patient case reports involving glucagon-like peptide 1 receptor agonists, with clinical events associated with their most probable reference times. We evaluated automated LLM timeline extraction against gold-standard timelines annotated by clinical domain experts, assessing how well systems recovered clinical events and their timings. The best-performing LLM produced high event coverage (GPT5; 0.871) and reliable temporal sequencing across symptoms (GPT5; 0.843), diagnoses, treatments, laboratory tests, and outcomes. As a downstream demonstration, time-to-event analyses in diabetes suggested lower risk of respiratory sequelae among GLP-1 users versus non-users (HR=0.259, p!0.05), consistent with prior reports of improved respiratory outcomes. Temporal annotations and code will be released upon acceptance.
- Open Access
Task-Adaptive Wasserstein Barycentric Fusion for Multimodal Medical Imaging
Multimodal medical image analysis requires fusion that accounts for task-dependent modality contributions while preserving shared and modality-specific information. To address these needs, we propose the Task-Adaptive Wasserstein Barycenter Variational Autoencoder ( Wℬ VAE-TA). The model forms a shared latent distribution as a Wasserstein barycenter of modality-specific encoding distributions. To adapt fusion to the task, we learn the modality weights jointly with the application objective while retaining closed-form aggregation of diagonal Gaussian distributions. To model modality-specific characteristics alongside the fused shared representations, the model uses multilevel decoding with modality-specific decoders and learnable modality embeddings. We evaluate Wℬ VAE-TA in brain tumor segmentation with complete and incomplete MRI inputs and in normative modeling with complete structural and diffusion-derived measurements. On BraTS 2018, the model achieves mean Dice scores of 62.31%, 78.83%, and 87.30% for enhancing tumor, tumor core, and whole tumor, respectively, averaged across all 15 modality combinations. Ablation studies demonstrate consistent gains from adaptive fusion and modality-specific conditioning. In normative modeling, the model achieves the highest estimated data log-likelihood among the evaluated methods on UK Biobank and ADNI, together with the highest ADNI significance ratio, precision, and balanced accuracy. These results support task-adaptive barycentric fusion with modality-specific conditioning across discriminative and generative medical imaging tasks.
Open AccessTask-Adaptive Wasserstein Barycentric Fusion for Multimodal Medical Imaging
Multimodal medical image analysis requires fusion that accounts for task-dependent modality contributions while preserving shared and modality-specific information. To address these needs, we propose the Task-Adaptive Wasserstein Barycenter Variational Autoencoder ( Wℬ VAE-TA). The model forms a shared latent distribution as a Wasserstein barycenter of modality-specific encoding distributions. To adapt fusion to the task, we learn the modality weights jointly with the application objective while retaining closed-form aggregation of diagonal Gaussian distributions. To model modality-specific characteristics alongside the fused shared representations, the model uses multilevel decoding with modality-specific decoders and learnable modality embeddings. We evaluate Wℬ VAE-TA in brain tumor segmentation with complete and incomplete MRI inputs and in normative modeling with complete structural and diffusion-derived measurements. On BraTS 2018, the model achieves mean Dice scores of 62.31%, 78.83%, and 87.30% for enhancing tumor, tumor core, and whole tumor, respectively, averaged across all 15 modality combinations. Ablation studies demonstrate consistent gains from adaptive fusion and modality-specific conditioning. In normative modeling, the model achieves the highest estimated data log-likelihood among the evaluated methods on UK Biobank and ADNI, together with the highest ADNI significance ratio, precision, and balanced accuracy. These results support task-adaptive barycentric fusion with modality-specific conditioning across discriminative and generative medical imaging tasks.
- Open Access
Multimodal normative modeling in Alzheimers Disease with introspective variational autoencoders
Normative modeling learns a healthy reference distribution and quantifies subject-specific deviations to capture heterogeneous disease effects. In Alzheimers disease (AD), multimodal neuroimaging offers complementary signals but VAE-based normative models often (i) fit the healthy reference distribution imperfectly, inflating false positives, and (ii) use posterior aggregation (e.g., PoE/MoE) that can yield weak multimodal fusion in the shared latent space. We propose mmSIVAE, a multimodal soft-introspective variational autoencoder combined with Mixture-of-Product-of-Experts (MOPOE) aggregation to improve reference fidelity and multimodal integration. We compute deviation scores in latent space and feature space as distances from the learned healthy distributions, and map statistically significant latent deviations to regional abnormalities for interpretability. On ADNI MRI regional volumes and amyloid PET SUVR, mmSIVAE improves reconstruction on held-out controls and produces more discriminative deviation scores for outlier detection than VAE baselines, with higher likelihood ratios and clearer separation between control and AD-spectrum cohorts. Deviation maps highlight region-level patterns aligned with established AD-related changes. More broadly, our results highlight the importance of training objectives that prioritize reference-distribution fidelity and robust multimodal posterior aggregation for normative modeling, with implications for deviation-based analysis across multimodal clinical data.
Open AccessMultimodal normative modeling in Alzheimers Disease with introspective variational autoencoders
Normative modeling learns a healthy reference distribution and quantifies subject-specific deviations to capture heterogeneous disease effects. In Alzheimers disease (AD), multimodal neuroimaging offers complementary signals but VAE-based normative models often (i) fit the healthy reference distribution imperfectly, inflating false positives, and (ii) use posterior aggregation (e.g., PoE/MoE) that can yield weak multimodal fusion in the shared latent space. We propose mmSIVAE, a multimodal soft-introspective variational autoencoder combined with Mixture-of-Product-of-Experts (MOPOE) aggregation to improve reference fidelity and multimodal integration. We compute deviation scores in latent space and feature space as distances from the learned healthy distributions, and map statistically significant latent deviations to regional abnormalities for interpretability. On ADNI MRI regional volumes and amyloid PET SUVR, mmSIVAE improves reconstruction on held-out controls and produces more discriminative deviation scores for outlier detection than VAE baselines, with higher likelihood ratios and clearer separation between control and AD-spectrum cohorts. Deviation maps highlight region-level patterns aligned with established AD-related changes. More broadly, our results highlight the importance of training objectives that prioritize reference-distribution fidelity and robust multimodal posterior aggregation for normative modeling, with implications for deviation-based analysis across multimodal clinical data.
- Open Access
Forecasting Clinical Risk from Textual Time Series: Structuring Narratives for Temporal AI in Healthcare
Clinical case reports encode temporal patient trajectories that are often underexploited by traditional machine learning methods relying on structured data. In this work, we introduce the forecasting problem from textual time series, where timestamped clinical findings–extracted via an LLM-assisted annotation pipeline–serve as the primary input for prediction. We systematically evaluate a diverse suite of models, including fine-tuned decoder-based large language models and encoder-based transformers, on tasks of event occurrence prediction, temporal ordering, and survival analysis. Our experiments reveal that encoder-based models consistently achieve higher F1 scores and superior temporal concordance for short- and long-horizon event forecasting, while fine-tuned masking approaches enhance ranking performance. In contrast, instruction-tuned decoder models demonstrate a relative advantage in survival analysis, especially in early prognosis settings. Our sensitivity analyses further demonstrate the importance of time ordering, which requires clinical time series construction, as compared to text ordering, the format of the text inputs that LLMs are classically trained on. This highlights the additional benefit that can be ascertained from time-ordered corpora, with implications for temporal tasks in the era of widespread LLM use.
Open AccessForecasting Clinical Risk from Textual Time Series: Structuring Narratives for Temporal AI in Healthcare
Clinical case reports encode temporal patient trajectories that are often underexploited by traditional machine learning methods relying on structured data. In this work, we introduce the forecasting problem from textual time series, where timestamped clinical findings–extracted via an LLM-assisted annotation pipeline–serve as the primary input for prediction. We systematically evaluate a diverse suite of models, including fine-tuned decoder-based large language models and encoder-based transformers, on tasks of event occurrence prediction, temporal ordering, and survival analysis. Our experiments reveal that encoder-based models consistently achieve higher F1 scores and superior temporal concordance for short- and long-horizon event forecasting, while fine-tuned masking approaches enhance ranking performance. In contrast, instruction-tuned decoder models demonstrate a relative advantage in survival analysis, especially in early prognosis settings. Our sensitivity analyses further demonstrate the importance of time ordering, which requires clinical time series construction, as compared to text ordering, the format of the text inputs that LLMs are classically trained on. This highlights the additional benefit that can be ascertained from time-ordered corpora, with implications for temporal tasks in the era of widespread LLM use.
- Open Access
Temporally annotated textual time series from PubMed Open Access clinical case reports
Understanding temporal dynamics in clinical narratives is essential for modeling patient trajectories, yet large-scale temporally annotated resources remain limited. We present PMOA–TTS, the first openly available dataset of 124,699 PubMed Open Access (PMOA) case reports, each converted into structured (event, time) timelines via a scalable LLM-based pipeline. Our approach combines heuristic filtering with Llama 3.3 to identify single-patient case reports, followed by prompt-driven extraction using Llama 3.3 and DeepSeek R1, resulting in over 5.6 million timestamped clinical events. To assess timeline quality, we evaluate against a clinician-curated reference set using three metrics: (i) event-level matching (80% match at a cosine similarity threshold of 0.1), (ii) temporal concordance (c-index > 0.90), and (iii) Area Under the Log-Time CDF (AULTC) for timestamp alignment. Corpus-level analysis shows wide diagnostic and demographic coverage. In a downstream survival prediction task, embeddings from extracted timelines achieve time-dependent concordance indices up to 0.82 ± 0.01, demonstrating the predictive value of temporally structured narratives. PMOA–TTS provides a scalable foundation for timeline extraction, temporal reasoning, and longitudinal modeling in biomedical NLP. The dataset is available at: https://huggingface.co/datasets/snoroozi/pmoa-tts .
Open AccessTemporally annotated textual time series from PubMed Open Access clinical case reports
Understanding temporal dynamics in clinical narratives is essential for modeling patient trajectories, yet large-scale temporally annotated resources remain limited. We present PMOA–TTS, the first openly available dataset of 124,699 PubMed Open Access (PMOA) case reports, each converted into structured (event, time) timelines via a scalable LLM-based pipeline. Our approach combines heuristic filtering with Llama 3.3 to identify single-patient case reports, followed by prompt-driven extraction using Llama 3.3 and DeepSeek R1, resulting in over 5.6 million timestamped clinical events. To assess timeline quality, we evaluate against a clinician-curated reference set using three metrics: (i) event-level matching (80% match at a cosine similarity threshold of 0.1), (ii) temporal concordance (c-index > 0.90), and (iii) Area Under the Log-Time CDF (AULTC) for timestamp alignment. Corpus-level analysis shows wide diagnostic and demographic coverage. In a downstream survival prediction task, embeddings from extracted timelines achieve time-dependent concordance indices up to 0.82 ± 0.01, demonstrating the predictive value of temporally structured narratives. PMOA–TTS provides a scalable foundation for timeline extraction, temporal reasoning, and longitudinal modeling in biomedical NLP. The dataset is available at: https://huggingface.co/datasets/snoroozi/pmoa-tts .
- Open Access
- 10 October 2025
- Published by Elsevier in Biomedical Signal Processing and Control
- Vol. 112
AgileFormer: Spatially agile and scalable transformer for medical image segmentation
- 10 October 2025
- Published by Elsevier in Biomedical Signal Processing and Control
- Vol. 112
Open AccessAgileFormer: Spatially agile and scalable transformer for medical image segmentation
- Open Access
- 15 April 2025
- Published by Wiley in Alzheimer's & Dementia
- Vol. 21 (4) , e70143
Analyzing heterogeneity in Alzheimer disease using multimodal normative modeling on imaging‐based ATN biomarkers
INTRODUCTION: Previous studies have applied normative modeling on a single neuroimaging modality to investigate Alzheimer disease (AD) heterogeneity. We employed a deep learning‐based multimodal normative framework to analyze individual‐level variation across ATN (amyloid‐tau‐neurodegeneration) imaging biomarkers. METHODS: We selected cross‐sectional discovery ( n = 665) and replication cohorts ( n = 430) with available T1‐weighted magnetic resonance imaging (MRI), amyloid, and tau positron emission tomography (PET). Normative modeling estimated individual‐level abnormal deviations in amyloid‐positive individuals compared to amyloid‐negative controls. Regional abnormality patterns were mapped at different clinical group levels to assess intra‐group heterogeneity. An individual‐level disease severity index (DSI) was calculated using both the spatial extent and magnitude of abnormal deviations across ATN. RESULTS: Greater intra‐group heterogeneity in ATN abnormality patterns was observed in more severe clinical stages of AD. Higher DSI was associated with worse cognitive function and increased risk of disease progression. DISCUSSION: Subject‐specific abnormality maps across ATN reveal the heterogeneous impact of AD on the brain. Highlights: Normative modeling examined AD heterogeneity across multimodal imaging biomarkers. Heterogeneity in spatial patterns of gray matter atrophy, amyloid, and tau burden. Higher within‐group heterogeneity for AD patients at advanced dementia stages. Patient‐specific metric summarized extent of neurodegeneration and neuropathology. Metric is a marker of poor brain health and can monitor risk of disease progression.
- 15 April 2025
- Published by Wiley in Alzheimer's & Dementia
- Vol. 21 (4) , e70143
Open AccessAnalyzing heterogeneity in Alzheimer disease using multimodal normative modeling on imaging‐based ATN biomarkers
INTRODUCTION: Previous studies have applied normative modeling on a single neuroimaging modality to investigate Alzheimer disease (AD) heterogeneity. We employed a deep learning‐based multimodal normative framework to analyze individual‐level variation across ATN (amyloid‐tau‐neurodegeneration) imaging biomarkers. METHODS: We selected cross‐sectional discovery ( n = 665) and replication cohorts ( n = 430) with available T1‐weighted magnetic resonance imaging (MRI), amyloid, and tau positron emission tomography (PET). Normative modeling estimated individual‐level abnormal deviations in amyloid‐positive individuals compared to amyloid‐negative controls. Regional abnormality patterns were mapped at different clinical group levels to assess intra‐group heterogeneity. An individual‐level disease severity index (DSI) was calculated using both the spatial extent and magnitude of abnormal deviations across ATN. RESULTS: Greater intra‐group heterogeneity in ATN abnormality patterns was observed in more severe clinical stages of AD. Higher DSI was associated with worse cognitive function and increased risk of disease progression. DISCUSSION: Subject‐specific abnormality maps across ATN reveal the heterogeneous impact of AD on the brain. Highlights: Normative modeling examined AD heterogeneity across multimodal imaging biomarkers. Heterogeneity in spatial patterns of gray matter atrophy, amyloid, and tau burden. Higher within‐group heterogeneity for AD patients at advanced dementia stages. Patient‐specific metric summarized extent of neurodegeneration and neuropathology. Metric is a marker of poor brain health and can monitor risk of disease progression.
- Open Access
- 15 April 2025
- Vol. 21 (4)
Analyzing heterogeneity in Alzheimer disease using multimodal normative modeling on imaging‐based ATN biomarkers
Previous studies have applied normative modeling on a single neuroimaging modality to investigate Alzheimer disease (AD) heterogeneity. We employed a deep learning‐based multimodal normative framework to analyze individual‐level variation across ATN (amyloid‐tau‐neurodegeneration) imaging biomarkers. We selected cross‐sectional discovery (n = 665) and replication cohorts (n = 430) with available T1‐weighted magnetic resonance imaging (MRI), amyloid, and tau positron emission tomography (PET). Normative modeling estimated individual‐level abnormal deviations in amyloid‐positive individuals compared to amyloid‐negative controls. Regional abnormality patterns were mapped at different clinical group levels to assess intra‐group heterogeneity. An individual‐level disease severity index (DSI) was calculated using both the spatial extent and magnitude of abnormal deviations across ATN. Greater intra‐group heterogeneity in ATN abnormality patterns was observed in more severe clinical stages of AD. Higher DSI was associated with worse cognitive function and increased risk of disease progression. Subject‐specific abnormality maps across ATN reveal the heterogeneous impact of AD on the brain. Normative modeling examined AD heterogeneity across multimodal imaging biomarkers. Heterogeneity in spatial patterns of gray matter atrophy, amyloid, and tau burden. Higher within‐group heterogeneity for AD patients at advanced dementia stages. Patient‐specific metric summarized extent of neurodegeneration and neuropathology. Metric is a marker of poor brain health and can monitor risk of disease progression.
- 15 April 2025
- Vol. 21 (4)
Open AccessAnalyzing heterogeneity in Alzheimer disease using multimodal normative modeling on imaging‐based ATN biomarkers
Previous studies have applied normative modeling on a single neuroimaging modality to investigate Alzheimer disease (AD) heterogeneity. We employed a deep learning‐based multimodal normative framework to analyze individual‐level variation across ATN (amyloid‐tau‐neurodegeneration) imaging biomarkers. We selected cross‐sectional discovery (n = 665) and replication cohorts (n = 430) with available T1‐weighted magnetic resonance imaging (MRI), amyloid, and tau positron emission tomography (PET). Normative modeling estimated individual‐level abnormal deviations in amyloid‐positive individuals compared to amyloid‐negative controls. Regional abnormality patterns were mapped at different clinical group levels to assess intra‐group heterogeneity. An individual‐level disease severity index (DSI) was calculated using both the spatial extent and magnitude of abnormal deviations across ATN. Greater intra‐group heterogeneity in ATN abnormality patterns was observed in more severe clinical stages of AD. Higher DSI was associated with worse cognitive function and increased risk of disease progression. Subject‐specific abnormality maps across ATN reveal the heterogeneous impact of AD on the brain. Normative modeling examined AD heterogeneity across multimodal imaging biomarkers. Heterogeneity in spatial patterns of gray matter atrophy, amyloid, and tau burden. Higher within‐group heterogeneity for AD patients at advanced dementia stages. Patient‐specific metric summarized extent of neurodegeneration and neuropathology. Metric is a marker of poor brain health and can monitor risk of disease progression.
- Open Access
- 11 April 2025
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 39 (19) , 20060-20068
Multimodal Variational Autoencoder: A Barycentric View
Multiple signal modalities, such as vision and sounds, are naturally present in real-world phenomena. Recently, there has been growing interest in learning generative models, in particular variational autoencoder (VAE), to for multimodal representation learning especially in the case of missing modalities. The primary goal of these models is to learn a modality-invariant and modality-specific representation that characterizes information across multiple modalities. Previous attempts at multimodal VAEs approach this mainly through the lens of experts, aggregating unimodal inference distributions with a product of experts (PoE), a mixture of experts (MoE), or a combination of both. In this paper, we provide an alternative generic and theoretical formulation of multimodal VAE through the lens of barycenter. We first show that PoE and MoE are specific instances of barycenters, derived by minimizing the asymmetric weighted KL divergence to unimodal inference distributions. Our novel formulation extends these two barycenters to a more flexible choice by considering different types of divergences. In particular, we explore the Wasserstein barycenter defined by the 2-Wasserstein distance, which better preserves the geometry of unimodal distributions by capturing both modality-specific and modality-invariant representations compared to KL divergence. Empirical studies on three multimodal benchmarks demonstrated the effectiveness of the proposed method.
- 11 April 2025
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 39 (19) , 20060-20068
Open AccessMultimodal Variational Autoencoder: A Barycentric View
Multiple signal modalities, such as vision and sounds, are naturally present in real-world phenomena. Recently, there has been growing interest in learning generative models, in particular variational autoencoder (VAE), to for multimodal representation learning especially in the case of missing modalities. The primary goal of these models is to learn a modality-invariant and modality-specific representation that characterizes information across multiple modalities. Previous attempts at multimodal VAEs approach this mainly through the lens of experts, aggregating unimodal inference distributions with a product of experts (PoE), a mixture of experts (MoE), or a combination of both. In this paper, we provide an alternative generic and theoretical formulation of multimodal VAE through the lens of barycenter. We first show that PoE and MoE are specific instances of barycenters, derived by minimizing the asymmetric weighted KL divergence to unimodal inference distributions. Our novel formulation extends these two barycenters to a more flexible choice by considering different types of divergences. In particular, we explore the Wasserstein barycenter defined by the 2-Wasserstein distance, which better preserves the geometry of unimodal distributions by capturing both modality-specific and modality-invariant representations compared to KL divergence. Empirical studies on three multimodal benchmarks demonstrated the effectiveness of the proposed method.
- Open Access
- 7 February 2025
- Published by Elsevier in American Journal of Transplantation
- Vol. 25 (6) , 1339-1349
Developing approaches to incorporate donor-lung computed tomography images into machine learning models to predict severe primary graft dysfunction after lung transplantation
- 7 February 2025
- Published by Elsevier in American Journal of Transplantation
- Vol. 25 (6) , 1339-1349
Open AccessDeveloping approaches to incorporate donor-lung computed tomography images into machine learning models to predict severe primary graft dysfunction after lung transplantation