Ambiguity-Aware Multi-Stage Cell-Type Annotation for Spatial Transcriptomics
Preprint
Abstract
Spatial transcriptomics enables characterization of cellular organization in intact tissue, but robust cell-type annotation remains challenging due to heterogeneous expression profiles, mixed populations, and transitional states. Existing methods often enforce a single label per cluster, obscuring biologically meaningful ambiguity and producing overconfident assignments. We propose an ambiguity-aware, multi-stage framework for spatial cell-type annotation. The method combines hybrid spatial–feature clustering with constrained language-model inference over curated label sets, and assigns confidence scores based on marker coverage, candidate separation, and entropy. Low-confidence clusters are selectively refined via local reclustering of ambiguous regions, while unresolved clusters are preserved as mixed rather than forcibly labeled. Applied to 10x Genomics Xenium spatial transcriptomics data from cholangiocarcinoma, the proposed refinement reduces cluster-level ambiguity from 16.1% to 2.27% and cell-level ambiguity from 18.4% to 0.86%, while improving confidence calibration. Spatial ablation confirms that topological integration resolves structural ambiguity over feature-only baselines, while constrained inference via a lightweight language model ensures scalable and biologically coherent annotations. These results highlight the importance of explicit ambiguity handling for reliable spatial annotation in heterogeneous tumors.Keywords
Funding Information
- National Center for Advancing Translational Sciences (NCATS), National Institutes of Health (UM1TR004539)
This publication has 11 references indexed in Scilit:
- Artificial Intelligence in genomics: a comprehensive survey of methods, resources, challenges, and prospectsBriefings in Bioinformatics, 2026
- ACID: Improving Ambiguous Cell Type Annotation in Spatial Transcriptomics2025
- High-precision cell-type mapping and annotation of single-cell spatial transcriptomics with STAMapperGenome Biology, 2025
- Evaluation of cell type annotation reliability using a large language model-based identifierCommunications Biology, 2025
- spaLLM: enhancing spatial domain analysis in multi-omics data through large language model integrationBriefings in Bioinformatics, 2025
- scGPT: toward building a foundation model for single-cell multi-omics using generative AINature Methods, 2024
- Spatially informed clustering, integration, and deconvolution of spatial transcriptomics with GraphSTNature Communications, 2023
- CellMarker 2.0: an updated database of manually curated cell markers in human/mouse and web tools based on scRNA-seq dataNucleic Acids Research, 2022
- SpaGCN: Integrating gene expression, spatial location and histology to identify spatial domains and spatially variable genes by graph convolutional networkNature Methods, 2021
- PanglaoDB: a web server for exploration of mouse and human single-cell RNA sequencing dataDatabase: The Journal of Biological Databases and Curation, 2019