Jianbing Zhang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Jianbing Zhang has a total of 59 co-authors. Here are the top co-authors by number of shared publications:
Chenyang Yan
National Key Laboratory for Novel Software Technology Nanjing University, Nanjing, Jiangsu, China
4 shared publications- Kanzhi Cheng
Nanjing University
4 shared publications - Fangzhi Xu
Xi'an Jiaotong University
3 shared publications - Hao Zhou
Institute of AI Industry Research(AIR), Tsinghua University, Beijing, China
3 shared publications - Mianzhi Pan
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, Jiangsu, China
3 shared publications Yougang Chu
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, Jiangsu, 210023, P. R. China
3 shared publications
Latest Publications
- Open Access
- 25 August 2026
- Published by MDPI AG in Preprints.org
Is Multimodal Speculative Decoding Ready for Diffusion-Based Parallel Drafting? A Survey and Empirical Diagnosis
Speculative decoding accelerates autoregressive generation by allowing a lightweight drafter to propose future tokens while a target model verifies them in parallel. Its lossless guarantee has motivated a line of work that pushes the drafter itself into parallel generation—the most recent enorous method is block-parallel generative approaches, including diffusion-based drafting methods such as DFlash and DSpark, achieving a 3.6x speedup on common daily chatting tasks. While this transition is well studied in text-only LLMs, its applicability to multimodal models remains an open question. Existing multimodal speculative decoding efforts focus on input compression, adapter alignment, candidate coverage, or modality-specific verification; however, block-parallel generative drafting remains largely unexplored. To bridge this gap, this paper combines a modality-centered survey with a cross-architecture empirical study to ask: Is multimodal speculative decoding ready for diffusion-based parallel drafting? In this survey, we systematically analyze a wide spectrum of multimodal models—spanning Vision-Language, Video-Language, Audio, and Vision-Language-Action (VLA) architectures—from the dual perspectives of drafting parallelism and cross-modal information interaction. We introduce a unified taxonomy that isolates drafter-side parallelism from orthogonal design choices such as tree construction and verification strategies. Furthermore, we provide a comprehensive empirical comparison of existing methods under varying degrees of parallelism across standardized multimodal benchmarks, including OCR, VQA, visual reasoning, and image captioning. Finally, we summarize the limitations of current approaches, discuss open challenges, and outline promising future directions for this rapidly evolving field.
- 25 August 2026
- Published by MDPI AG in Preprints.org
Open AccessIs Multimodal Speculative Decoding Ready for Diffusion-Based Parallel Drafting? A Survey and Empirical Diagnosis
Speculative decoding accelerates autoregressive generation by allowing a lightweight drafter to propose future tokens while a target model verifies them in parallel. Its lossless guarantee has motivated a line of work that pushes the drafter itself into parallel generation—the most recent enorous method is block-parallel generative approaches, including diffusion-based drafting methods such as DFlash and DSpark, achieving a 3.6x speedup on common daily chatting tasks. While this transition is well studied in text-only LLMs, its applicability to multimodal models remains an open question. Existing multimodal speculative decoding efforts focus on input compression, adapter alignment, candidate coverage, or modality-specific verification; however, block-parallel generative drafting remains largely unexplored. To bridge this gap, this paper combines a modality-centered survey with a cross-architecture empirical study to ask: Is multimodal speculative decoding ready for diffusion-based parallel drafting? In this survey, we systematically analyze a wide spectrum of multimodal models—spanning Vision-Language, Video-Language, Audio, and Vision-Language-Action (VLA) architectures—from the dual perspectives of drafting parallelism and cross-modal information interaction. We introduce a unified taxonomy that isolates drafter-side parallelism from orthogonal design choices such as tree construction and verification strategies. Furthermore, we provide a comprehensive empirical comparison of existing methods under varying degrees of parallelism across standardized multimodal benchmarks, including OCR, VQA, visual reasoning, and image captioning. Finally, we summarize the limitations of current approaches, discuss open challenges, and outline promising future directions for this rapidly evolving field.
- Open Access
Strict OOD Antigen-to-Antibody Retrieval with CDR-Aware Slot Late Interaction
Antigen-specific antibody retrieval aims to rank candidate antibodies for a target antigen, providing an early virtual-screening step before structural modeling or experimental validation. Existing sequence-based antibody-antigen interaction studies often formulate the problem as pairwise binding prediction, and random or non-clustered evaluations can over-estimate generalization when related antigens appear across training and test data. We study a strict antigen-cluster out-of-distribution (OOD) retrieval setting in which test antigens come from sequence clusters unseen during training. This setting is difficult because binding is driven by local epitope-CDR complementarity, while available databases mainly contain observed positive complexes and lack reliable negative labels for unlabeled candidates. We propose Ab-CASLR, an antibody CDR-aware slot late-interaction retriever that encodes antigens with ESM-2, encodes antibodies with IgBert, constrains antibody-side latent slots to complementarity-determining regions (CDRs), and scores local slot compatibility instead of single-vector global similarity. On a strict OOD benchmark with 849 antigen queries and 869 candidate antibodies, the model achieves 7.42% Hits@10, outperforming k-mer homology transfer at 5.53% Hits@10 and yielding 6.28-fold enrichment over exact random screening at K = 10. Ablations and diagnostics show that CDR-constrained antibody slots remain diverse, whereas antigen-side latent slots collapse into similar summaries. These results support CDR-aware local antibody representation as a useful inductive bias for early binder recovery under strict OOD evaluation, while antigen-side epitope grounding remains unresolved.
Open AccessStrict OOD Antigen-to-Antibody Retrieval with CDR-Aware Slot Late Interaction
Antigen-specific antibody retrieval aims to rank candidate antibodies for a target antigen, providing an early virtual-screening step before structural modeling or experimental validation. Existing sequence-based antibody-antigen interaction studies often formulate the problem as pairwise binding prediction, and random or non-clustered evaluations can over-estimate generalization when related antigens appear across training and test data. We study a strict antigen-cluster out-of-distribution (OOD) retrieval setting in which test antigens come from sequence clusters unseen during training. This setting is difficult because binding is driven by local epitope-CDR complementarity, while available databases mainly contain observed positive complexes and lack reliable negative labels for unlabeled candidates. We propose Ab-CASLR, an antibody CDR-aware slot late-interaction retriever that encodes antigens with ESM-2, encodes antibodies with IgBert, constrains antibody-side latent slots to complementarity-determining regions (CDRs), and scores local slot compatibility instead of single-vector global similarity. On a strict OOD benchmark with 849 antigen queries and 869 candidate antibodies, the model achieves 7.42% Hits@10, outperforming k-mer homology transfer at 5.53% Hits@10 and yielding 6.28-fold enrichment over exact random screening at K = 10. Ablations and diagnostics show that CDR-constrained antibody slots remain diverse, whereas antigen-side latent slots collapse into similar summaries. These results support CDR-aware local antibody representation as a useful inductive bias for early binder recovery under strict OOD evaluation, while antigen-side epitope grounding remains unresolved.
- Open Access
Robust Conditional Diffusion with Noisy Templates for Antibody Sequence–Structure Design
Antibodies specifically recognize antigens and play a central role in therapeutic discovery. Designing antibodies for a given antigen remains challenging because antigen–antibody complex data are limited, whereas the sequence and conformational spaces of complementarity-determining regions (CDRs) are large. Retrieved CDR templates from databases or candidate libraries can narrow the design space and improve controllability, but retrieval for novel antigens is often sparse and imperfect; treating retrieved templates as hard conditions can bias the denoising process and cause negative transfer. To address this problem, we propose Robust Conditional Diffusion with Noisy Templates for antibody sequence–structure design (NT-ABDiff), a joint diffusion framework that treats candidate CDR-only templates as optional and potentially unreliable conditions. NT-ABDiff uses reliability-aware template modulation to estimate the context-conditioned usefulness of each candidate and to adaptively reweight and fuse multiple templates during conditioning. We further train the model with mixed-quality and corrupted templates as conditional perturbation regularization, encouraging the denoiser to exploit informative templates while remaining stable when templates are uninformative. Experiments under controlled template shifts and a train-set retrieval evaluation show that NT-ABDiff improves CDR-H3 sequence recovery and structural accuracy over strong baselines, while retaining robustness to missing, mismatched, and corrupted templates. Under a stringent random-template CDR-H3 evaluation, NT-ABDiff improves amino-acid recovery (AAR) from 30.03% to 39.47% and reduces RMSD from 3.160 to 2.915 Å; with train-set retrieval candidates, it achieves 39.50% AAR and 2.76 Å RMSD. Code, processed splits, configuration files, and evaluation scripts are available at https://github.com/ShiDeng7rz/NT-ABDiff .
Open AccessRobust Conditional Diffusion with Noisy Templates for Antibody Sequence–Structure Design
Antibodies specifically recognize antigens and play a central role in therapeutic discovery. Designing antibodies for a given antigen remains challenging because antigen–antibody complex data are limited, whereas the sequence and conformational spaces of complementarity-determining regions (CDRs) are large. Retrieved CDR templates from databases or candidate libraries can narrow the design space and improve controllability, but retrieval for novel antigens is often sparse and imperfect; treating retrieved templates as hard conditions can bias the denoising process and cause negative transfer. To address this problem, we propose Robust Conditional Diffusion with Noisy Templates for antibody sequence–structure design (NT-ABDiff), a joint diffusion framework that treats candidate CDR-only templates as optional and potentially unreliable conditions. NT-ABDiff uses reliability-aware template modulation to estimate the context-conditioned usefulness of each candidate and to adaptively reweight and fuse multiple templates during conditioning. We further train the model with mixed-quality and corrupted templates as conditional perturbation regularization, encouraging the denoiser to exploit informative templates while remaining stable when templates are uninformative. Experiments under controlled template shifts and a train-set retrieval evaluation show that NT-ABDiff improves CDR-H3 sequence recovery and structural accuracy over strong baselines, while retaining robustness to missing, mismatched, and corrupted templates. Under a stringent random-template CDR-H3 evaluation, NT-ABDiff improves amino-acid recovery (AAR) from 30.03% to 39.47% and reduces RMSD from 3.160 to 2.915 Å; with train-set retrieval candidates, it achieves 39.50% AAR and 2.76 Å RMSD. Code, processed splits, configuration files, and evaluation scripts are available at https://github.com/ShiDeng7rz/NT-ABDiff .
- Open Access
- 4 September 2025
- Published by Royal Society of Chemistry (RSC) in Digital Discovery
- Vol. 4 (10) , 2942-2953
Generative AI-powered inverse design for tailored narrowband molecular emitters
We introduce an AI-driven molecular generation framework. It leverages efficient Markov chain Monte Carlo sampling and multi-objective optimization, enabling rapid, tailored design of narrowband emitters across the desired color spectrum.
- 4 September 2025
- Published by Royal Society of Chemistry (RSC) in Digital Discovery
- Vol. 4 (10) , 2942-2953
Open AccessGenerative AI-powered inverse design for tailored narrowband molecular emitters
We introduce an AI-driven molecular generation framework. It leverages efficient Markov chain Monte Carlo sampling and multi-objective optimization, enabling rapid, tailored design of narrowband emitters across the desired color spectrum.
- 1 September 2025
- Published by Springer Nature in Journal of Computer Science and Technology
- Vol. 40 (5) , 1209-1219
KnowBench: Evaluating the Knowledge Alignment on Large Visual Language Models
- 1 September 2025
- Published by Springer Nature in Journal of Computer Science and Technology
- Vol. 40 (5) , 1209-1219
KnowBench: Evaluating the Knowledge Alignment on Large Visual Language Models
- Open Access
CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era
Image captioning has been a longstanding challenge in vision-language research. With the rise of LLMs, modern Vision-Language Models (VLMs) generate detailed and comprehensive image descriptions. However, benchmarking the quality of such captions remains unresolved. This paper addresses two key questions: (1) How well do current VLMs actually perform on image captioning, particularly compared to humans? We built CapArena, a platform with over 6000 pairwise caption battles and high-quality human preference votes. Our arena-style evaluation marks a milestone, showing that leading models like GPT-4o achieve or even surpass human performance, while most open-source models lag behind. (2) Can automated metrics reliably assess detailed caption quality? Using human annotations from CapArena, we evaluate traditional and recent captioning metrics, as well as VLM-as-a-Judge. Our analysis reveals that while some metrics (e.g., METEOR) show decent caption-level agreement with humans, their systematic biases lead to inconsistencies in model ranking. In contrast, VLM-as-a-Judge demonstrates robust discernment at both the caption and model levels. Building on these insights, we release CapArena-Auto, an accurate and efficient automated benchmark for detailed captioning, achieving 94.3% correlation with human rankings at just $4 per test. Data and resources will be open-sourced at https://caparena.github.io.
Open AccessCapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era
Image captioning has been a longstanding challenge in vision-language research. With the rise of LLMs, modern Vision-Language Models (VLMs) generate detailed and comprehensive image descriptions. However, benchmarking the quality of such captions remains unresolved. This paper addresses two key questions: (1) How well do current VLMs actually perform on image captioning, particularly compared to humans? We built CapArena, a platform with over 6000 pairwise caption battles and high-quality human preference votes. Our arena-style evaluation marks a milestone, showing that leading models like GPT-4o achieve or even surpass human performance, while most open-source models lag behind. (2) Can automated metrics reliably assess detailed caption quality? Using human annotations from CapArena, we evaluate traditional and recent captioning metrics, as well as VLM-as-a-Judge. Our analysis reveals that while some metrics (e.g., METEOR) show decent caption-level agreement with humans, their systematic biases lead to inconsistencies in model ranking. In contrast, VLM-as-a-Judge demonstrates robust discernment at both the caption and model levels. Building on these insights, we release CapArena-Auto, an accurate and efficient automated benchmark for detailed captioning, achieving 94.3% correlation with human rankings at just $4 per test. Data and resources will be open-sourced at https://caparena.github.io.
- Top Cited PapersOpen Access
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 14077-14094
CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era
Kanzhi Cheng, Wenpo Song, Jiaxin Fan, Zheng Ma, Qiushi Sun, Fangzhi Xu, Chenyang Yan, Nuo Chen, Jianbing Zhang, Jiajun Chen. Findings of the Association for Computational Linguistics: ACL 2025. 2025.
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 14077-14094
Top Cited PapersOpen AccessCapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era
Kanzhi Cheng, Wenpo Song, Jiaxin Fan, Zheng Ma, Qiushi Sun, Fangzhi Xu, Chenyang Yan, Nuo Chen, Jianbing Zhang, Jiajun Chen. Findings of the Association for Computational Linguistics: ACL 2025. 2025.
- Top Cited PapersOpen Access
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 8876-8892
Vision-Language Models Can Self-Improve Reasoning via Reflection
Kanzhi Cheng, Li YanTao, Fangzhi Xu, Jianbing Zhang, Hao Zhou, Yang Liu. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 8876-8892
Top Cited PapersOpen AccessVision-Language Models Can Self-Improve Reasoning via Reflection
Kanzhi Cheng, Li YanTao, Fangzhi Xu, Jianbing Zhang, Hao Zhou, Yang Liu. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.
- Open Access
- 27 November 2024
- Published by American Chemical Society (ACS) in ChemRxiv
Generative AI-Powered Inverse Design for TailoredNarrowband Molecular Emitters
In organic displays, developing molecules that produce a broad color gamut with exceptional color purity is of critical importance. AI-assisted molecular screening can expedite the design process of emission molecules. However, the efficiency of current methodologies is constrained by their limited candidate pools and poor hit rates. Here we present MEMOS, a cutting-edge molecular generation framework that, through Markov molecular sampling techniques, facilitates the targeted inverse design of molecules across a nearly boundless chemical space, tailored to emit the narrow spectral bands associated with desired colors. Notably, by employing a self-improving iterative process, MEMOS achieves an impressive hit rate of up to 80%. Our method showcases the pioneering capability to rapidly navigate through millions of molecular possibilities, efficiently pinpointing thousands of high-potential candidates within a 24-hour period. This breakthrough accelerates the design of novel organic luminescent materials, setting the stage for the advancement of the next generation of high-quality organic displays.
- 27 November 2024
- Published by American Chemical Society (ACS) in ChemRxiv
Open AccessGenerative AI-Powered Inverse Design for TailoredNarrowband Molecular Emitters
In organic displays, developing molecules that produce a broad color gamut with exceptional color purity is of critical importance. AI-assisted molecular screening can expedite the design process of emission molecules. However, the efficiency of current methodologies is constrained by their limited candidate pools and poor hit rates. Here we present MEMOS, a cutting-edge molecular generation framework that, through Markov molecular sampling techniques, facilitates the targeted inverse design of molecules across a nearly boundless chemical space, tailored to emit the narrow spectral bands associated with desired colors. Notably, by employing a self-improving iterative process, MEMOS achieves an impressive hit rate of up to 80%. Our method showcases the pioneering capability to rapidly navigate through millions of molecular possibilities, efficiently pinpointing thousands of high-potential candidates within a 24-hour period. This breakthrough accelerates the design of novel organic luminescent materials, setting the stage for the advancement of the next generation of high-quality organic displays.
- Open Access
SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents
Graphical User Interface (GUI) agents are designed to automate complex tasks on digital devices, such as smartphones and desktops. Most existing GUI agents interact with the environment through extracted structured data, which can be notably lengthy (e.g., HTML) and occasionally inaccessible (e.g., on desktops). To alleviate this issue, we propose a novel visual GUI agent -- SeeClick, which only relies on screenshots for task automation. In our preliminary study, we have discovered a key challenge in developing visual GUI agents: GUI grounding -- the capacity to accurately locate screen elements based on instructions. To tackle this challenge, we propose to enhance SeeClick with GUI grounding pre-training and devise a method to automate the curation of GUI grounding data. Along with the efforts above, we have also created ScreenSpot, the first realistic GUI grounding benchmark that encompasses mobile, desktop, and web environments. After pre-training, SeeClick demonstrates significant improvement in ScreenSpot over various baselines. Moreover, comprehensive evaluations on three widely used benchmarks consistently support our finding that advancements in GUI grounding directly correlate with enhanced performance in downstream GUI agent tasks. The model, data and code are available at https://github.com/njucckevin/SeeClick.
Open AccessSeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents
Graphical User Interface (GUI) agents are designed to automate complex tasks on digital devices, such as smartphones and desktops. Most existing GUI agents interact with the environment through extracted structured data, which can be notably lengthy (e.g., HTML) and occasionally inaccessible (e.g., on desktops). To alleviate this issue, we propose a novel visual GUI agent -- SeeClick, which only relies on screenshots for task automation. In our preliminary study, we have discovered a key challenge in developing visual GUI agents: GUI grounding -- the capacity to accurately locate screen elements based on instructions. To tackle this challenge, we propose to enhance SeeClick with GUI grounding pre-training and devise a method to automate the curation of GUI grounding data. Along with the efforts above, we have also created ScreenSpot, the first realistic GUI grounding benchmark that encompasses mobile, desktop, and web environments. After pre-training, SeeClick demonstrates significant improvement in ScreenSpot over various baselines. Moreover, comprehensive evaluations on three widely used benchmarks consistently support our finding that advancements in GUI grounding directly correlate with enhanced performance in downstream GUI agent tasks. The model, data and code are available at https://github.com/njucckevin/SeeClick.