TZTao Zhang

Tao Zhang

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Tao Zhang has a total of 64 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Top Cited Papers

    Sa2VA: Marrying SAM2 With MLLM for Dense Grounded Understanding of Images and Videos

    by Haobo Yuan, Xiangtai Li, Tao Zhang, Yueyi Sun, Zilong Huang, Shilin Xu, Shunping Ji, Yunhai Tong, Lu Qi, Jiashi Feng, et al.

    This work presents Sa2VA, the first comprehensive, unified model for dense grounded understanding of both images and videos. Unlike existing multi-modal large language models, which are often limited to specific modalities and tasks, Sa2VA supports a wide range of image and video tasks, including referring segmentation and conversation, with minimal single-stage instruction tuning. Sa2VA combines SAM-2, a foundation video segmentation model, with MLLM, advanced vision-language models, and unifies text, image, and video into a shared LLM token space. Using the LLM, Sa2VA generates instruction tokens that guide SAM-2 in producing precise masks, enabling a grounded, multi-modal understanding of both static and dynamic visual content. Additionally, we introduce Ref-SAV, an auto-labeled dataset containing over 72k object expressions in complex video scenes, designed to boost model performance. We also manually validate 2k video objects in the Ref-SAV dataset to benchmark referring video object segmentation in complex environments. Experiments show that Sa2VA achieves strong performance across multiple tasks, particularly in referring video object segmentation, highlighting its potential for complex real-world applications. In addition, Sa2VA can be easily extended to various MLLMs, including Qwen-VL and Intern-VL, which can be updated with a rapid process in current open-source MLLMs. Code and models have been provided to the community.


  • Open Access

    Visual Tokenizers: A Comprehensive Survey

    by Qingyu Shi, Haobo Yuan, Yikang Zhou, Xiu Li, Jason Li, Size Wu, Haochen Wang, Ye Tian, Tao Zhang, Jinbin Bai, et al.

    Visual tokenization bridges the gap between high-dimensional visual data and sequence modeling by transforming images, videos, and 3D content into compact token sequences. Recent advances in Multimodal Large Language Models (LLMs) have further underscored the critical role of visual tokenizers. Despite this rapid progress, the literature in this area remains fragmented. This work addresses this gap by presenting a comprehensive taxonomy of visual tokenizers. We review the evolution of task-specific tokenizers along two complementary dimensions: high-level tokenizers that emphasize semantic representations and low-level tokenizers that focus on pixel reconstruction and compression. Motivated by the emergence of Multimodal Large Language Models capable of supporting both understanding and generation tasks, we highlight recent advances in unified tokenizers. These models aim to integrate semantic abstraction with fine-grained visual details within a single representation. We categorize unified tokenizers into single-encoder and dual-encoder architectures and further organize them according to their semantic preservation strategies and fusion mechanisms. We also summarize empirical results on downstream understanding and generation benchmarks, comparing unified tokenizers with both individual task-specific models and combinations of separate tokenizers. Finally, we discuss open challenges and future directions, including the trade-off between semantic abstraction and reconstruction fidelity, scalable token vocabularies, and extensions to video and 3D domains. We further examine the evolving role of visual tokens in next-generation multimodal systems. We maintain a curated list of related works at https://github.com/Shi-qingyu/Awesome-Visual-Tokenizer.

  • High-Quality Entity Segmentation and Grounding

    by Lu Qi, Yi-Wen Chen, Tao Zhang, Xiangtai Li, Xu Yang, Bo Du, Ming-Hsuan Yang

    Class-agnostic segmentation is of great importance for generalizing the localization of semantically-consistent regions from limited training data to unseen domains and categories. It has benefited numerous applications, such as human-interaction-assisted image editing. Despite their success, existing class-agnostic methods (e.g., SAM2 and entity segmentation) still struggle with boundary-accurate masks and semantic understanding. To address these issues, we propose ESG, a pipeline for high-quality entity segmentation and grounding supported by a new dataset EntitySeg. At first, the proposed dataset naming EntitySeg contains images spanning various image domains and entities, along with plentiful high-resolution images and high-quality mask annotations for training and testing. Then, the ESG mainly consists of two modules: CropFormer for high-quality entity segmentation whereas GELLA for accurate noun extraction from sentences and semantic matching between language and visual regions. Unlike existing grounding methods that jointly train a segmentation and a large language model, ESG adopts a two-stage decoupled design, preserving high-quality masks and grounding robustness without the trade-offs often introduced by joint training. CropFormer ensures high-quality entity segmentation results, which can then be encoded into the GELLA model for effective grounding. Extensive experimental results demonstrate the effectiveness of our proposed pipeline across five tasks, including entity segmentation, panoptic segmentation, open-vocabulary segmentation, referring segmentation, and panoptic localized narratives. Furthermore, GELLA module of ESG pipeline is highly flexible and capable of processing mask inputs from any segmentation framework, thanks to its lightweight colormap/vision encoder, language/mask decoder, and association module. The entity segmentation dataset and grounding code will be released at https://github.com/qqlu/Entity.

  • Open Access

    MapSAM2: Adapting SAM2 for Automatic Segmentation of Historical Map Images and Time Series

    by Xue Xia, Randall Balestriero, Tao Zhang, Yixin Zhou, Andrew Ding, Dev Saini, Lorenz Hurni

    Historical maps are unique and valuable archives that document geographic features across different time periods. However, automated analysis of historical map images remains a significant challenge due to their wide stylistic variability and the scarcity of annotated training data. Constructing linked spatio-temporal datasets from historical map time series is even more time-consuming and labor-intensive, as it requires synthesizing information from multiple maps. Such datasets are essential for applications such as dating buildings, analyzing the development of road networks and settlements, studying environmental changes etc. We present MapSAM2, a unified framework for automatically segmenting both historical map images and time series. Built on a visual foundation model, MapSAM2 adapts to diverse segmentation tasks with few-shot fine-tuning. Our key innovation is to treat both historical map images and time series as videos. For images, we process a set of tiles as a video, enabling the memory attention mechanism to incorporate contextual cues from similar tiles, leading to improved geometric accuracy, particularly for areal features. For time series, we introduce the annotated Siegfried Building Time Series Dataset and, to reduce annotation costs, propose generating pseudo time series from single-year maps by simulating common temporal transformations. Experimental results show that MapSAM2 learns temporal associations effectively and can accurately segment and link buildings in time series under limited supervision or using pseudo videos. We release both our dataset and code to support future research.


  • Open Access

    DenseScan: Advancing 3D Scene Understanding with 2D Dense Annotation

    by Zirui Wang, Tao Zhang

    3D understanding is a key capability for real-world AI assistance. High-quality data plays an important role in driving the development of the 3D understanding community. Current 3D scene understanding datasets often provide geometric and instance-level information, yet they lack the rich semantic annotations necessary for nuanced visual-language tasks.In this work, we introduce DenseScan, a novel dataset with detailed multi-level descriptions generated by an automated pipeline leveraging multi-view 2D images and multimodal large language models (MLLMs). Our approach enables dense captioning of scene elements, ensuring comprehensive object-level descriptions that capture context-sensitive details. Furthermore, we extend these annotations through scenario-based question generation, producing high-level queries that integrate object properties, spatial relationships, and scene context. By coupling geometric detail with semantic richness, DenseScan broadens the range of downstream tasks, from detailed visual-language navigation to interactive question answering. Experimental results demonstrate that our method significantly enhances object-level understanding and question-answering performance in 3D environments compared to traditional annotation pipelines. We release both the annotated dataset and our annotation pipeline to facilitate future research and applications in robotics, augmented reality, and beyond. Through DenseScan, we aim to catalyze new avenues in 3D scene understanding, allowing researchers and practitioners to tackle the complexities of real-world environments with richer, more contextually aware annotations.

  • P2PFormerV2: Improving Primitive-Based Regular Building Contour Extraction Methods via Contour Feature Enhancement

    by Wenling Yu, Tao Zhang, Shunping Ji, Kun Zhang, Bo Liu, Hua Liu, Jianya Gong

    Deep learning methods have been widely used to map building contours automatically in high-resolution remote sensing images over recent years. However, most deep learning-based methods still require complex post-processing to generate regular contours. P2PFormer is an advanced method that can directly obtain the positions and sequences of general geometric primitives like points, lines, and angles (corners) of a building instance without complex post-processing. Nevertheless, due to the inherent limitations of ROI-Align, P2PFormer faces significant challenges in primitive detection. ROI-Align utilizes a uniform sampling approach for feature extraction, it inevitably introduces a high proportion of invalid sampling points into the extracted features. Resulting in missed and false detections of building primitives, ultimately affecting the accuracy of building contour extraction. To address these issues, we propose P2PFormerV2, which introduces a contour feature enhancer to improve P2PFormer. The contour feature enhancer increases the proportion of valid feature sampling points and enhances the extraction of contour-aware features, significantly improving the accuracy of primitive segmentation. This enhancer comprises three key components: the sparse feature extractor, the dense feature extractor, and the feature fusion module. The sparse feature extractor optimizes the sampling strategy to increase the proportion of valid feature sampling points; the dense feature extractor generates rich contour features and provides additional supervision signals; the feature fusion module integrates the outputs of the first two components to further enhance the extraction of contour features. Experimental results demonstrate that P2PFormerV2 achieves average precisions (AP) of 74.7%, 79.6%, and 64.2% in the WHU, CrowdAI, and WHU-Mix datasets, respectively, significantly outperforming the original P2PFormer and other existing advanced methods. Our findings about the shortcomings of ROI-Align and the importance of improving the effective feature extraction provides insights for future building extraction research.


  • Open Access

    SVAC: Scaling Is All You Need For Referring Video Object Segmentation

    by Li Zhang, Haoxiang Gao, Zhihao Zhang, Luoxiao Huang, Tao Zhang

    Referring Video Object Segmentation (RVOS) aims to segment target objects in video sequences based on natural language descriptions. While recent advances in Multi-modal Large Language Models (MLLMs) have improved RVOS performance through enhanced text-video understanding, several challenges remain, including insufficient exploitation of MLLMs' prior knowledge, prohibitive computational and memory costs for long-duration videos, and inadequate handling of complex temporal dynamics. In this work, we propose SVAC, a unified model that improves RVOS by scaling up input frames and segmentation tokens to enhance video-language interaction and segmentation precision. To address the resulting computational challenges, SVAC incorporates the Anchor-Based Spatio-Temporal Compression (ASTC) module to compress visual tokens while preserving essential spatio-temporal structure. Moreover, the Clip-Specific Allocation (CSA) strategy is introduced to better handle dynamic object behaviors across video clips. Experimental results demonstrate that SVAC achieves state-of-the-art performance on multiple RVOS benchmarks with competitive efficiency. Our code is available at https://github.com/lizhang1998/SVAC.

  • Top Cited Papers
    Open Access

    Point Cloud Mamba: Point Cloud Learning via State Space Model

    by Tao Zhang, Haobo Yuan, Lu Qi, Jiangning Zhang, Qianyu Zhou, Shunping Ji, Shuicheng Yan, Xiangtai Li

    Recently, state space models have exhibited strong global modeling capabilities and linear computational complexity in contrast to transformers. This research focuses on applying such architecture to more efficiently and effectively model point cloud data globally with linear computational complexity. In particular, for the first time, we demonstrate that Mamba-based point cloud methods can outperform previous methods based on transformer or multi-layer perceptrons (MLPs). To enable Mamba to process 3-D point cloud data more effectively, we propose a novel Consistent Traverse Serialization method to convert point clouds into 1-D point sequences while ensuring that neighboring points in the sequence are also spatially adjacent. Consistent Traverse Serialization yields six variants by permuting the order of x, y, and z coordinates, and the synergistic use of these variants aids Mamba in comprehensively observing point cloud data. Furthermore, to assist Mamba in handling point sequences with different orders more effectively, we introduce point prompts to inform Mamba of the sequence’s arrangement rules. Finally, we propose positional encoding based on spatial coordinate mapping to inject positional information into point cloud sequences more effectively. Point Cloud Mamba surpasses the state-of-the-art (SOTA) point-based method PointNeXt and achieves new SOTA performance on the ScanObjectNN, ModelNet40, ShapeNetPart, and S3DIS datasets. It is worth mentioning that when using a more powerful local feature extraction module, our PCM achieves 79.6 mIoU on S3DIS, significantly surpassing the previous SOTA models, DeLA and PTv3, by 5.5 mIoU and 4.9 mIoU, respectively.


  • Open Access

    CD-STMamba: Toward Remote Sensing Image Change Detection With Spatio-Temporal Interaction Mamba Model

    by Shanwei Liu, Shuaipeng Wang, Wei Zhang, Tao Zhang, Mingming Xu, Muhammad Yasir, Shiqing Wei

    Change detection (CD) is a critical Earth observation task. Convolutional neural network (CNN) and Transformer have demonstrated their superior performance in CD tasks. However, the limitations of the limited receptive field of CNN and the high-computational complexity of Transformer remain. Recently, the Mamba architecture, based on state-space models, has demonstrated strong global receptive field capabilities and implements linear time complexity in computational processes. While some researchers have incorporated it into CD tasks, most have neglected the effective application of the Mamba selective scanning algorithm for modeling bitemporal image dependencies, resulting in suboptimal feature learning from bitemporal images. In this article, we propose a CD Mamba model (CD-STMamba), which can efficiently encode and decode bitemporal images interactively from multiple dimensions, thus enabling more accurate CD. Specifically, we propose a spatio-temporal interaction module (STIM), which can interact with bitemporal image features in multiple dimensions and fit with the Mamba architecture, allowing it to fully learn the global contextual information of the bitemporal input image. We also introduce a decoding block called the CD block, which can be fully decoded to learn multiple spatio-temporal relationships based on the characteristics of STIM. This block employs multiple change visual state space blocks internally to decode different spatio-temporal interactions and utilizes the change attention module to capture change features comprehensively for more accurate CD. The proposed CD-STMamba achieved state-of-the-art intersection over union (IoU) on three datasets, Wuhan University Building Change Detection Dataset (91.29% ), Sun Yat-Sen University Change Detection (73.45% ), and Change Detection Dataset (95.56% ).


  • Top Cited Papers

    DVIS++: Improved Decoupled Framework for Universal Video Segmentation

    by Tao Zhang, Xingye Tian, Yikang Zhou, Shunping Ji, Xuebo Wang, Xin Tao, Yuan Zhang, Pengfei Wan, Zhongyuan Wang, Yu Wu

    We present the Decoupled VIdeo Segmentation (DVIS) framework, a novel approach for the challenging task of universal video segmentation, including video instance segmentation (VIS), video semantic segmentation (VSS), and video panoptic segmentation (VPS). Unlike previous methods that model video segmentation in an end-to-end manner, our approach decouples video segmentation into three cascaded sub-tasks: segmentation, tracking, and refinement. This decoupling design allows for simpler and more effective modeling of the spatio-temporal representations of objects, especially in complex scenes and long videos. Accordingly, we introduce two novel components: the referring tracker and the temporal refiner. These components track objects frame by frame and model spatio-temporal representations based on pre-aligned features. To improve the tracking capability of DVIS, we propose a denoising training strategy and introduce contrastive learning, resulting in a more robust framework named DVIS++. The proposed decoupled framework efficiently handles universal and open-vocabulary object representations, allowing DVIS++ to conduct universal and open-vocabulary video segmentation. We conduct extensive experiments on six mainstream benchmarks, including the VIS, VSS, and VPS datasets. Using a unified architecture, DVIS++ significantly outperforms state-of-the-art specialized methods on these benchmarks in closed- and open-vocabulary settings.