Junnan Zhu
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Junnan Zhu has a total of 83 co-authors. Here are the top co-authors by number of shared publications:
Chengqing Zong
9 shared publications- Chengqing Zong
Institute of Automation, Chinese Academy of Sciences
9 shared publications Jiajun Zhang
National Laboratory of Pattern Recognition, Institute of Automation, CAS, School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing Academy of Artificial Intelligence, Beijing, China
9 shared publicationsHaoran Li
JD AI Research
7 shared publications- Yu Zhou
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China
7 shared publications Yu Zhou
National Laboratory of Pattern Recognition, Institute of Automation, CAS, School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing Fanyu Technology Co., Ltd, Beijing, China
5 shared publications
Latest Publications
- Open Access
Structure-aware deep learning predicts influenza antigenicity and guides vaccine strain recommendation
The continuous accumulation of genetic mutations in influenza A viruses (IAVs) drives antigenic drift, necessitating precise antigenic prediction for optimal vaccine strain selection. While sequence-based methods have advanced antigenic surveillance, they neglect the three-dimensional structural context that fundamentally dictates viral antigenicity. Here, we introduce Vir3D, which leverages ESMFold-derived structural information from amino acid sequences to precisely predict viral antigenicity and guide vaccine strain selection. Across both human H3 and highly pathogenic avian H5 subtypes, Vir3D not only accurately discriminates antigenic variants and infers pairwise antigenic distances, but also mechanistically delineates key structural residues driving viral immune evasion. In a decade-long retrospective analysis, Vir3D-prioritized vaccines consistently achieve broader antigenic coverage of circulating strains than World Health Organization (WHO) recommendations. Crucially, Vir3D successfully predicts that the emerging U.S. dairy cattle H5N1 virus (TX/24) remains antigenically stable relative to clade 2.3.4.4b vaccine strains, and subsequent wet-laboratory validation of hemagglutination inhibition (HI) assays definitively corroborates this finding. Overall, Vir3D establishes a powerful, structure-driven framework for proactive influenza surveillance and pandemic preparedness.
Open AccessStructure-aware deep learning predicts influenza antigenicity and guides vaccine strain recommendation
The continuous accumulation of genetic mutations in influenza A viruses (IAVs) drives antigenic drift, necessitating precise antigenic prediction for optimal vaccine strain selection. While sequence-based methods have advanced antigenic surveillance, they neglect the three-dimensional structural context that fundamentally dictates viral antigenicity. Here, we introduce Vir3D, which leverages ESMFold-derived structural information from amino acid sequences to precisely predict viral antigenicity and guide vaccine strain selection. Across both human H3 and highly pathogenic avian H5 subtypes, Vir3D not only accurately discriminates antigenic variants and infers pairwise antigenic distances, but also mechanistically delineates key structural residues driving viral immune evasion. In a decade-long retrospective analysis, Vir3D-prioritized vaccines consistently achieve broader antigenic coverage of circulating strains than World Health Organization (WHO) recommendations. Crucially, Vir3D successfully predicts that the emerging U.S. dairy cattle H5N1 virus (TX/24) remains antigenically stable relative to clade 2.3.4.4b vaccine strains, and subsequent wet-laboratory validation of hemagglutination inhibition (HI) assays definitively corroborates this finding. Overall, Vir3D establishes a powerful, structure-driven framework for proactive influenza surveillance and pandemic preparedness.
- Open Access
- 9 June 2026
- Published by Wiley in Advanced Science
- Vol. 13 (49) , e75969
Accurately Deciphering Tissue Heterogeneity From Spatial Multi‐Modal and Multi‐Omics With STransformer
Advances in spatially resolved technologies enable the simultaneous acquisition of diverse data modalities within a tissue slice while preserving critical spatial context, which presents unprecedented opportunities to decipher intricate tissue heterogeneity. However, existing computational approaches lack the intrinsic flexibility to universally process both spatial multi‐modal and multi‐omics data. Here, we introduce STransformer, a unified deep learning framework designed to seamlessly accommodate a comprehensive landscape of spatial data. By simultaneously capturing short‐range cellular interactions and tissue‐wide semantic patterns, it extracts robust representations to accurately dissect complex tissue heterogeneity. Systematic evaluations across diverse species, tissue types, and data modalities highlight its profound versatility. For spatial multi‐modal data, STransformer delineates intricate anatomical structures in the human cortex, uncovers pathological mechanisms in Alzheimer's disease, and characterizes dynamic spatiotemporal developmental trajectories during chicken cardiogenesis. Scaling to spatial multi‐omics data, STransformer synergizes spatial transcriptomic and proteomic profiles to decipher intricate immune microenvironments within the human tonsil, and jointly analyzes spatial epigenomic and transcriptomic data to infer regulatory mechanisms in the mouse embryonic brain. Consequently, STransformer serves as a highly versatile and robust analytical framework for advancing our understanding of tissue heterogeneity and disease pathogenesis.
- 9 June 2026
- Published by Wiley in Advanced Science
- Vol. 13 (49) , e75969
Open AccessAccurately Deciphering Tissue Heterogeneity From Spatial Multi‐Modal and Multi‐Omics With STransformer
Advances in spatially resolved technologies enable the simultaneous acquisition of diverse data modalities within a tissue slice while preserving critical spatial context, which presents unprecedented opportunities to decipher intricate tissue heterogeneity. However, existing computational approaches lack the intrinsic flexibility to universally process both spatial multi‐modal and multi‐omics data. Here, we introduce STransformer, a unified deep learning framework designed to seamlessly accommodate a comprehensive landscape of spatial data. By simultaneously capturing short‐range cellular interactions and tissue‐wide semantic patterns, it extracts robust representations to accurately dissect complex tissue heterogeneity. Systematic evaluations across diverse species, tissue types, and data modalities highlight its profound versatility. For spatial multi‐modal data, STransformer delineates intricate anatomical structures in the human cortex, uncovers pathological mechanisms in Alzheimer's disease, and characterizes dynamic spatiotemporal developmental trajectories during chicken cardiogenesis. Scaling to spatial multi‐omics data, STransformer synergizes spatial transcriptomic and proteomic profiles to decipher intricate immune microenvironments within the human tonsil, and jointly analyzes spatial epigenomic and transcriptomic data to infer regulatory mechanisms in the mouse embryonic brain. Consequently, STransformer serves as a highly versatile and robust analytical framework for advancing our understanding of tissue heterogeneity and disease pathogenesis.
- Open Access
FocalOrder: Focal Preference Optimization for Reading Order Detection
Reading order detection is the foundation of document understanding. Most existing methods rely on uniform supervision, implicitly assuming a constant difficulty distribution across layout regions. In this work, we challenge this assumption by revealing a critical flaw: \textbf{Positional Disparity}, a phenomenon where models demonstrate mastery over the deterministic start and end regions but suffer a performance collapse in the complex intermediate sections. This degradation arises because standard training allows the massive volume of easy patterns to drown out the learning signals from difficult layouts. To address this, we propose \textbf{FocalOrder}, a framework driven by \textbf{Focal Preference Optimization (FPO)}. Specifically, FocalOrder employs adaptive difficulty discovery with exponential moving average mechanism to dynamically pinpoint hard-to-learn transitions, while introducing a difficulty-calibrated pairwise ranking objective to enforce global logical consistency. Extensive experiments demonstrate that FocalOrder establishes new state-of-the-art results on OmniDocBench v1.0 and Comp-HRDoc. Our compact model not only outperforms competitive specialized baselines but also significantly surpasses large-scale general VLMs. These results demonstrate that aligning the optimization with intrinsic structural ambiguity of documents is critical for mastering complex document structures.
Open AccessFocalOrder: Focal Preference Optimization for Reading Order Detection
Reading order detection is the foundation of document understanding. Most existing methods rely on uniform supervision, implicitly assuming a constant difficulty distribution across layout regions. In this work, we challenge this assumption by revealing a critical flaw: \textbf{Positional Disparity}, a phenomenon where models demonstrate mastery over the deterministic start and end regions but suffer a performance collapse in the complex intermediate sections. This degradation arises because standard training allows the massive volume of easy patterns to drown out the learning signals from difficult layouts. To address this, we propose \textbf{FocalOrder}, a framework driven by \textbf{Focal Preference Optimization (FPO)}. Specifically, FocalOrder employs adaptive difficulty discovery with exponential moving average mechanism to dynamically pinpoint hard-to-learn transitions, while introducing a difficulty-calibrated pairwise ranking objective to enforce global logical consistency. Extensive experiments demonstrate that FocalOrder establishes new state-of-the-art results on OmniDocBench v1.0 and Comp-HRDoc. Our compact model not only outperforms competitive specialized baselines but also significantly surpasses large-scale general VLMs. These results demonstrate that aligning the optimization with intrinsic structural ambiguity of documents is critical for mastering complex document structures.
- Open Access
PARL: Position-Aware Relation Learning Network for Document Layout Analysis
Document layout analysis aims to detect and categorize structural elements (e.g., titles, tables, figures) in scanned or digital documents. Popular methods often rely on high-quality Optical Character Recognition (OCR) to merge visual features with extracted text. This dependency introduces two major drawbacks: propagation of text recognition errors and substantial computational overhead, limiting the robustness and practical applicability of multimodal approaches. In contrast to the prevailing multimodal trend, we argue that effective layout analysis depends not on text-visual fusion, but on a deep understanding of documents' intrinsic visual structure. To this end, we propose PARL (Position-Aware Relation Learning Network), a novel OCR-free, vision-only framework that models layout through positional sensitivity and relational structure. Specifically, we first introduce a Bidirectional Spatial Position-Guided Deformable Attention module to embed explicit positional dependencies among layout elements directly into visual features. Second, we design a Graph Refinement Classifier (GRC) to refine predictions by modeling contextual relationships through a dynamically constructed layout graph. Extensive experiments show PARL achieves state-of-the-art results. It establishes a new benchmark for vision-only methods on DocLayNet and, notably, surpasses even strong multimodal models on M6Doc. Crucially, PARL (65M) is highly efficient, using roughly four times fewer parameters than large multimodal models (256M), demonstrating that sophisticated visual structure modeling can be both more efficient and robust than multimodal fusion.
Open AccessPARL: Position-Aware Relation Learning Network for Document Layout Analysis
Document layout analysis aims to detect and categorize structural elements (e.g., titles, tables, figures) in scanned or digital documents. Popular methods often rely on high-quality Optical Character Recognition (OCR) to merge visual features with extracted text. This dependency introduces two major drawbacks: propagation of text recognition errors and substantial computational overhead, limiting the robustness and practical applicability of multimodal approaches. In contrast to the prevailing multimodal trend, we argue that effective layout analysis depends not on text-visual fusion, but on a deep understanding of documents' intrinsic visual structure. To this end, we propose PARL (Position-Aware Relation Learning Network), a novel OCR-free, vision-only framework that models layout through positional sensitivity and relational structure. Specifically, we first introduce a Bidirectional Spatial Position-Guided Deformable Attention module to embed explicit positional dependencies among layout elements directly into visual features. Second, we design a Graph Refinement Classifier (GRC) to refine predictions by modeling contextual relationships through a dynamically constructed layout graph. Extensive experiments show PARL achieves state-of-the-art results. It establishes a new benchmark for vision-only methods on DocLayNet and, notably, surpasses even strong multimodal models on M6Doc. Crucially, PARL (65M) is highly efficient, using roughly four times fewer parameters than large multimodal models (256M), demonstrating that sophisticated visual structure modeling can be both more efficient and robust than multimodal fusion.
- Open Access
BayesRAG: Probabilistic Mutual Evidence Corroboration for Multimodal Retrieval-Augmented Generation
Retrieval-Augmented Generation (RAG) has become a pivotal paradigm for Large Language Models (LLMs), yet current approaches struggle with visually rich documents by treating text and images as isolated retrieval targets. Existing methods relying solely on cosine similarity often fail to capture the semantic reinforcement provided by cross-modal alignment and layout-induced coherence. To address these limitations, we propose BayesRAG, a novel multimodal retrieval framework grounded in Bayesian inference and Dempster-Shafer evidence theory. Unlike traditional approaches that rank candidates strictly by similarity, BayesRAG models the intrinsic consistency of retrieved candidates across modalities as probabilistic evidence to refine retrieval confidence. Specifically, our method computes the posterior association probability for combinations of multimodal retrieval results, prioritizing text-image pairs that mutually corroborate each other in terms of both semantics and layout. Extensive experiments demonstrate that BayesRAG significantly outperforms state-of-the-art (SOTA) methods on challenging multimodal benchmarks. This study establishes a new paradigm for multimodal retrieval fusion that effectively resolves the isolation of heterogeneous modalities through an evidence fusion mechanism and enhances the robustness of retrieval outcomes. Our code is available at https://github.com/TioeAre/BayesRAG.
Open AccessBayesRAG: Probabilistic Mutual Evidence Corroboration for Multimodal Retrieval-Augmented Generation
Retrieval-Augmented Generation (RAG) has become a pivotal paradigm for Large Language Models (LLMs), yet current approaches struggle with visually rich documents by treating text and images as isolated retrieval targets. Existing methods relying solely on cosine similarity often fail to capture the semantic reinforcement provided by cross-modal alignment and layout-induced coherence. To address these limitations, we propose BayesRAG, a novel multimodal retrieval framework grounded in Bayesian inference and Dempster-Shafer evidence theory. Unlike traditional approaches that rank candidates strictly by similarity, BayesRAG models the intrinsic consistency of retrieved candidates across modalities as probabilistic evidence to refine retrieval confidence. Specifically, our method computes the posterior association probability for combinations of multimodal retrieval results, prioritizing text-image pairs that mutually corroborate each other in terms of both semantics and layout. Extensive experiments demonstrate that BayesRAG significantly outperforms state-of-the-art (SOTA) methods on challenging multimodal benchmarks. This study establishes a new paradigm for multimodal retrieval fusion that effectively resolves the isolation of heterogeneous modalities through an evidence fusion mechanism and enhances the robustness of retrieval outcomes. Our code is available at https://github.com/TioeAre/BayesRAG.
- Open Access
ChartReasoner: Code-Driven Modality Bridging for Long-Chain Reasoning in Chart Question Answering
Recently, large language models have shown remarkable reasoning capabilities through long-chain reasoning before responding. However, how to extend this capability to visual reasoning tasks remains an open challenge. Existing multimodal reasoning approaches transfer such visual reasoning task into textual reasoning task via several image-to-text conversions, which often lose critical structural and semantic information embedded in visualizations, especially for tasks like chart question answering that require a large amount of visual details. To bridge this gap, we propose ChartReasoner, a code-driven novel two-stage framework designed to enable precise, interpretable reasoning over charts. We first train a high-fidelity model to convert diverse chart images into structured ECharts codes, preserving both layout and data semantics as lossless as possible. Then, we design a general chart reasoning data synthesis pipeline, which leverages this pretrained transport model to automatically and scalably generate chart reasoning trajectories and utilizes a code validator to filter out low-quality samples. Finally, we train the final multimodal model using a combination of supervised fine-tuning and reinforcement learning on our synthesized chart reasoning dataset and experimental results on four public benchmarks clearly demonstrate the effectiveness of our proposed ChartReasoner. It can preserve the original details of the charts as much as possible and perform comparably with state-of-the-art open-source models while using fewer parameters, approaching the performance of proprietary systems like GPT-4o in out-of-domain settings.
Open AccessChartReasoner: Code-Driven Modality Bridging for Long-Chain Reasoning in Chart Question Answering
Recently, large language models have shown remarkable reasoning capabilities through long-chain reasoning before responding. However, how to extend this capability to visual reasoning tasks remains an open challenge. Existing multimodal reasoning approaches transfer such visual reasoning task into textual reasoning task via several image-to-text conversions, which often lose critical structural and semantic information embedded in visualizations, especially for tasks like chart question answering that require a large amount of visual details. To bridge this gap, we propose ChartReasoner, a code-driven novel two-stage framework designed to enable precise, interpretable reasoning over charts. We first train a high-fidelity model to convert diverse chart images into structured ECharts codes, preserving both layout and data semantics as lossless as possible. Then, we design a general chart reasoning data synthesis pipeline, which leverages this pretrained transport model to automatically and scalably generate chart reasoning trajectories and utilizes a code validator to filter out low-quality samples. Finally, we train the final multimodal model using a combination of supervised fine-tuning and reinforcement learning on our synthesized chart reasoning dataset and experimental results on four public benchmarks clearly demonstrate the effectiveness of our proposed ChartReasoner. It can preserve the original details of the charts as much as possible and perform comparably with state-of-the-art open-source models while using fewer parameters, approaching the performance of proprietary systems like GPT-4o in out-of-domain settings.
- Open Access
ChartMind: A Comprehensive Benchmark for Complex Real-world Multimodal Chart Question Answering
Chart question answering (CQA) has become a critical multimodal task for evaluating the reasoning capabilities of vision-language models. While early approaches have shown promising performance by focusing on visual features or leveraging large-scale pre-training, most existing evaluations rely on rigid output formats and objective metrics, thus ignoring the complex, real-world demands of practical chart analysis. In this paper, we introduce ChartMind, a new benchmark designed for complex CQA tasks in real-world settings. ChartMind covers seven task categories, incorporates multilingual contexts, supports open-domain textual outputs, and accommodates diverse chart formats, bridging the gap between real-world applications and traditional academic benchmarks. Furthermore, we propose a context-aware yet model-agnostic framework, ChartLLM, that focuses on extracting key contextual elements, reducing noise, and enhancing the reasoning accuracy of multimodal large language models. Extensive evaluations on ChartMind and three representative public benchmarks with 14 mainstream multimodal models show our framework significantly outperforms the previous three common CQA paradigms: instruction-following, OCR-enhanced, and chain-of-thought, highlighting the importance of flexible chart understanding for real-world CQA. These findings suggest new directions for developing more robust chart reasoning in future research.
Open AccessChartMind: A Comprehensive Benchmark for Complex Real-world Multimodal Chart Question Answering
Chart question answering (CQA) has become a critical multimodal task for evaluating the reasoning capabilities of vision-language models. While early approaches have shown promising performance by focusing on visual features or leveraging large-scale pre-training, most existing evaluations rely on rigid output formats and objective metrics, thus ignoring the complex, real-world demands of practical chart analysis. In this paper, we introduce ChartMind, a new benchmark designed for complex CQA tasks in real-world settings. ChartMind covers seven task categories, incorporates multilingual contexts, supports open-domain textual outputs, and accommodates diverse chart formats, bridging the gap between real-world applications and traditional academic benchmarks. Furthermore, we propose a context-aware yet model-agnostic framework, ChartLLM, that focuses on extracting key contextual elements, reducing noise, and enhancing the reasoning accuracy of multimodal large language models. Extensive evaluations on ChartMind and three representative public benchmarks with 14 mainstream multimodal models show our framework significantly outperforms the previous three common CQA paradigms: instruction-following, OCR-enhanced, and chain-of-thought, highlighting the importance of flexible chart understanding for real-world CQA. These findings suggest new directions for developing more robust chart reasoning in future research.
- Open Access
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 7137-7157
TableEval: A Real-World Benchmark for Complex, Multilingual, and Multi-Structured Table Question Answering
Junnan Zhu, Jingyi Wang, Bohan Yu, Xiaoyu Wu, Junbo Li, Lei Wang, Nan Xu. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 7137-7157
Open AccessTableEval: A Real-World Benchmark for Complex, Multilingual, and Multi-Structured Table Question Answering
Junnan Zhu, Jingyi Wang, Bohan Yu, Xiaoyu Wu, Junbo Li, Lei Wang, Nan Xu. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.
- Open Access
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 4555-4569
ChartMind: A Comprehensive Benchmark for Complex Real-world Multimodal Chart Question Answering
Jingxuan Wei, Nan Xu, Junnan Zhu, Haoyanni, Gaowei Wu, Qi Chen, Bihui Yu, Lei Wang. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 4555-4569
Open AccessChartMind: A Comprehensive Benchmark for Complex Real-world Multimodal Chart Question Answering
Jingxuan Wei, Nan Xu, Junnan Zhu, Haoyanni, Gaowei Wu, Qi Chen, Bihui Yu, Lei Wang. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.
- Open Access
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 11755-11771
TROVE: A Challenge for Fine-Grained Text Provenance via Source Sentence Tracing and Relationship Classification
Junnan Zhu, Min Xiao, Yining Wang, Feifei Zhai, Yu Zhou, Chengqing Zong. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 11755-11771
Open AccessTROVE: A Challenge for Fine-Grained Text Provenance via Source Sentence Tracing and Relationship Classification
Junnan Zhu, Min Xiao, Yining Wang, Feifei Zhai, Yu Zhou, Chengqing Zong. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.