Xiaoliang Zhang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Xiaoliang Zhang has a total of 23 co-authors. Here are the top co-authors by number of shared publications:
- Fanman Meng
School of Information and Communication Engineering, University of Electronic Science and Technology of China, Chengdu, China
4 shared publications - Hongliang Li
University of Electronic Science and Technology of China
4 shared publications - Haoyang Cheng
School of Information and Communication Engineering, University of Electronic Science and Technology of China, Chengdu, China
3 shared publications - Heqian Qiu
University of Electronic Science and Technology of China
3 shared publications - Qingbo Wu
University of Electronic Science and Technology of China
3 shared publications - Zichen Song
School of Information and Communication Engineering, University of Electronic Science and Technology of China, Chengdu, China
2 shared publications
Latest Publications
- Open Access
- 23 February 2024
- Published by Association for Computing Machinery (ACM)
- p. 39-46
Dual Mask Branches for Instance Segmentation
Instance segmentation is an emerging and challenging task. A lot of previous methods have focused on this field and made great progress. However, complex background and mutual occlusion between instances still hinder the development of instance segmentation. In this paper, we aim to solve foreground confusion for instance segmentation. Based on Mask R-CNN, we first add an additional mask branch, which is parallel to the existing mask branch and expected to predict non-target foreground pixels within the predicted bounding box. To differentiate the two mask branches, the original mask branch is called Target Mask Branch while the additional mask branch is called Non-Target Mask Branch. Besides, in order to enhance the feature interaction and differences between the target mask branch and the non-target mask branch, we design a foreground confusion processing module to connect the dual mask branches. In this way, our method can distinguish the target instance and non-target foreground pixels within the corresponding predicted bounding box effectively, and obtain more accurate segmentation masks. Furthermore, a feature distinguish module is proposed to enhance the differences between features from the dual mask branches. We demonstrate our method's competitiveness on MS COCO dataset, which outperforms previous state-of-the-art instance segmentation methods. In addition, we also add our method into other instance segmentation methods, and the experiment results show our method still can obtain stable gains in performance.
- 23 February 2024
- Published by Association for Computing Machinery (ACM)
- p. 39-46
Open AccessDual Mask Branches for Instance Segmentation
Instance segmentation is an emerging and challenging task. A lot of previous methods have focused on this field and made great progress. However, complex background and mutual occlusion between instances still hinder the development of instance segmentation. In this paper, we aim to solve foreground confusion for instance segmentation. Based on Mask R-CNN, we first add an additional mask branch, which is parallel to the existing mask branch and expected to predict non-target foreground pixels within the predicted bounding box. To differentiate the two mask branches, the original mask branch is called Target Mask Branch while the additional mask branch is called Non-Target Mask Branch. Besides, in order to enhance the feature interaction and differences between the target mask branch and the non-target mask branch, we design a foreground confusion processing module to connect the dual mask branches. In this way, our method can distinguish the target instance and non-target foreground pixels within the corresponding predicted bounding box effectively, and obtain more accurate segmentation masks. Furthermore, a feature distinguish module is proposed to enhance the differences between features from the dual mask branches. We demonstrate our method's competitiveness on MS COCO dataset, which outperforms previous state-of-the-art instance segmentation methods. In addition, we also add our method into other instance segmentation methods, and the experiment results show our method still can obtain stable gains in performance.
- Open Access
Scale-Hybrid Group Distillation with Knowledge Disentangling for Continual Semantic Segmentation
Continual semantic segmentation (CSS) aims to learn new tasks sequentially and extract object(s) and stuff represented by pixel-level maps of new categories while preserving the original segmentation capabilities even when the old class data is absent. Current CSS methods typically preserve the capacities of segmenting old classes via knowledge distillation, which encounters the limitations of insufficient utilization of the semantic knowledge, i.e., only distilling the last layer of the feature encoder, and the semantic shift of background caused by directly distilling the entire feature map of the decoder. In this paper, we propose a novel CCS method based on scale-hybrid distillation and knowledge disentangling to address these limitations. Firstly, we propose a scale-hybrid group semantic distillation (SGD) method for encoding, which transfers the multi-scale knowledge from the old model’s feature encoder with group pooling refinement to improve the stability of new models. Then, the knowledge disentangling distillation (KDD) method for decoding is proposed to distillate feature maps with the guidance of the old class regions and reduce incorrect guides from old models towards better plasticity. Extensive experiments are conducted on the Pascal VOC and ADE20K datasets. Competitive performance compared with other state-of-the-art methods demonstrates the effectiveness of our proposed method.
Open AccessScale-Hybrid Group Distillation with Knowledge Disentangling for Continual Semantic Segmentation
Continual semantic segmentation (CSS) aims to learn new tasks sequentially and extract object(s) and stuff represented by pixel-level maps of new categories while preserving the original segmentation capabilities even when the old class data is absent. Current CSS methods typically preserve the capacities of segmenting old classes via knowledge distillation, which encounters the limitations of insufficient utilization of the semantic knowledge, i.e., only distilling the last layer of the feature encoder, and the semantic shift of background caused by directly distilling the entire feature map of the decoder. In this paper, we propose a novel CCS method based on scale-hybrid distillation and knowledge disentangling to address these limitations. Firstly, we propose a scale-hybrid group semantic distillation (SGD) method for encoding, which transfers the multi-scale knowledge from the old model’s feature encoder with group pooling refinement to improve the stability of new models. Then, the knowledge disentangling distillation (KDD) method for decoding is proposed to distillate feature maps with the guidance of the old class regions and reduce incorrect guides from old models towards better plasticity. Extensive experiments are conducted on the Pascal VOC and ADE20K datasets. Competitive performance compared with other state-of-the-art methods demonstrates the effectiveness of our proposed method.
- 3 May 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 33 (11) , 6924-6938
Disturbed Augmentation Invariance for Unsupervised Visual Representation Learning
Contrastive learning has gained great prominence recently, which achieves excellent performance by simple augmentation invariance. However, the simple contrastive pairs suffer from lacking of diversity due to the mechanical augmentation strategies. In this paper, we propose Disturbed Augmentation Invariance (DAI for abbreviation), which constructs disturbed contrastive pairs by generating appropriate disturbed views for each augmented view in the feature space to increase the diversity. In practice, we establish a multivariate normal distribution for each augmented view, whose mean is corresponding augmented view and covariance matrix is estimated from its nearest neighbors in the dataset. Then we sample random vectors from this distribution as the disturbed views to construct disturbed contrastive pairs. In order to avoid extra computational cost with the increase of disturbed contrastive pairs, we utilize an upper bound of the trivial disturbed augmentation invariance loss to construct the DAI loss. In addition, we propose Bottleneck version of Disturbed Augmentation Invariance (BDAI for abbreviation) inspired by the Information Bottleneck principle, which further refines the extracted information and learns a compact representation by additionally increasing the variance of the original contrastive pair. In order to make BDAI work effectively, we design a statistical strategy to control the balance between the amount of the information shared by all disturbed contrastive pairs and the compactness of the representation. Our approach gets a consistent improvement over the popular contrastive learning methods on a variety of downstream tasks, e.g. image classification, object detection and instance segmentation.
- 3 May 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 33 (11) , 6924-6938
Disturbed Augmentation Invariance for Unsupervised Visual Representation Learning
Contrastive learning has gained great prominence recently, which achieves excellent performance by simple augmentation invariance. However, the simple contrastive pairs suffer from lacking of diversity due to the mechanical augmentation strategies. In this paper, we propose Disturbed Augmentation Invariance (DAI for abbreviation), which constructs disturbed contrastive pairs by generating appropriate disturbed views for each augmented view in the feature space to increase the diversity. In practice, we establish a multivariate normal distribution for each augmented view, whose mean is corresponding augmented view and covariance matrix is estimated from its nearest neighbors in the dataset. Then we sample random vectors from this distribution as the disturbed views to construct disturbed contrastive pairs. In order to avoid extra computational cost with the increase of disturbed contrastive pairs, we utilize an upper bound of the trivial disturbed augmentation invariance loss to construct the DAI loss. In addition, we propose Bottleneck version of Disturbed Augmentation Invariance (BDAI for abbreviation) inspired by the Information Bottleneck principle, which further refines the extracted information and learns a compact representation by additionally increasing the variance of the original contrastive pair. In order to make BDAI work effectively, we design a statistical strategy to control the balance between the amount of the information shared by all disturbed contrastive pairs and the compactness of the representation. Our approach gets a consistent improvement over the popular contrastive learning methods on a variety of downstream tasks, e.g. image classification, object detection and instance segmentation.
- 24 February 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 32 (10577149) , 1613-1626
Unsupervised Visual Representation Learning via Multi-Dimensional Relationship Alignment
Recently, contrastive learning based on augmentation invariance and instance discrimination has made great achievements, owing to its excellent ability to learn beneficial representations without any manual annotations. However, the natural similarity among instances conflicts with instance discrimination which treats each instance as a unique individual. In order to explore the natural relationship among instances and integrate it into contrastive learning, we propose a novel approach in this paper, Relationship Alignment (RA for abbreviation), which forces different augmented views of current batch instances to main a consistent relationship with other instances. In order to perform RA effectively in existing contrastive learning framework, we design an alternating optimization algorithm where the relationship exploration step and alignment step are optimized respectively. In addition, we add an equilibrium constraint for RA to avoid the degenerate solution, and introduce the expansion handler to make it approximately satisfied in practice. In order to better capture the complex relationship among instances, we additionally propose Multi-Dimensional Relationship Alignment (MDRA for abbreviation), which aims to explore the relationship from multiple dimensions. In practice, we decompose the final high-dimensional feature space into a cartesian product of several low-dimensional subspaces and perform RA in each subspace respectively. We validate the effectiveness of our approach on multiple self-supervised learning benchmarks and get consistent improvements compared with current popular contrastive learning methods. On the most commonly used ImageNet linear evaluation protocol, our RA obtains significant improvements over other methods, our MDRA gets further improvements based on RA to achieve the best performance. The source code of our approach will be released soon.
- 24 February 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 32 (10577149) , 1613-1626
Unsupervised Visual Representation Learning via Multi-Dimensional Relationship Alignment
Recently, contrastive learning based on augmentation invariance and instance discrimination has made great achievements, owing to its excellent ability to learn beneficial representations without any manual annotations. However, the natural similarity among instances conflicts with instance discrimination which treats each instance as a unique individual. In order to explore the natural relationship among instances and integrate it into contrastive learning, we propose a novel approach in this paper, Relationship Alignment (RA for abbreviation), which forces different augmented views of current batch instances to main a consistent relationship with other instances. In order to perform RA effectively in existing contrastive learning framework, we design an alternating optimization algorithm where the relationship exploration step and alignment step are optimized respectively. In addition, we add an equilibrium constraint for RA to avoid the degenerate solution, and introduce the expansion handler to make it approximately satisfied in practice. In order to better capture the complex relationship among instances, we additionally propose Multi-Dimensional Relationship Alignment (MDRA for abbreviation), which aims to explore the relationship from multiple dimensions. In practice, we decompose the final high-dimensional feature space into a cartesian product of several low-dimensional subspaces and perform RA in each subspace respectively. We validate the effectiveness of our approach on multiple self-supervised learning benchmarks and get consistent improvements compared with current popular contrastive learning methods. On the most commonly used ImageNet linear evaluation protocol, our RA obtains significant improvements over other methods, our MDRA gets further improvements based on RA to achieve the best performance. The source code of our approach will be released soon.
- Open Access
- 23 December 2022
- Published by Association for Computing Machinery (ACM)
- p. 7-12
Three-Dimensional Offset Vector for Panoptic Segmentation
Panoptic segmentation has become a research hotspot in the field of computer vision. In this paper, we propose a novel and efficient method for panoptic segmentation. Based on semantic segmentation architecture DeepLabv3+, we predict a three-dimensional offset vector for each pixel, and produce a unified panoptic segmentation mask by combining semantic prediction. Previous bottom-up panoptic segmentation methods only consider offset vector in a horizontal plane, i.e. a two-dimensional space. In contrast, we consider offset vector in both horizontal plane and vertical direction, and predict an offset vector in three-dimensional space for each pixel. Offset vector in vertical direction is used to determine whether a pixel belongs to a foreground object, while offset vector in horizontal plane is used to determine which foreground object the foreground pixel belongs to. In this way, we can predict offset vectors for both background and foreground object pixels at the same time, and effectively model foreground pixels clustering process. During inference, we can produce a unified panoptic segmentation mask by combining the foreground pixels clustering result and semantic prediction. We demonstrate our method’s effectiveness on Cityscapes dataset, and obtain competitive results.
- 23 December 2022
- Published by Association for Computing Machinery (ACM)
- p. 7-12
Open AccessThree-Dimensional Offset Vector for Panoptic Segmentation
Panoptic segmentation has become a research hotspot in the field of computer vision. In this paper, we propose a novel and efficient method for panoptic segmentation. Based on semantic segmentation architecture DeepLabv3+, we predict a three-dimensional offset vector for each pixel, and produce a unified panoptic segmentation mask by combining semantic prediction. Previous bottom-up panoptic segmentation methods only consider offset vector in a horizontal plane, i.e. a two-dimensional space. In contrast, we consider offset vector in both horizontal plane and vertical direction, and predict an offset vector in three-dimensional space for each pixel. Offset vector in vertical direction is used to determine whether a pixel belongs to a foreground object, while offset vector in horizontal plane is used to determine which foreground object the foreground pixel belongs to. In this way, we can predict offset vectors for both background and foreground object pixels at the same time, and effectively model foreground pixels clustering process. During inference, we can produce a unified panoptic segmentation mask by combining the foreground pixels clustering result and semantic prediction. We demonstrate our method’s effectiveness on Cityscapes dataset, and obtain competitive results.
- 20 July 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 25 (15209210) , 5400-5412
What Happens in Crowd Scenes: A New Dataset About Crowd Scenes for Image Captioning
Making machines endowed with eyes and brains to effectively understand and analyze crowd scenes is of paramount importance for building a smart city to serve people. This is of far-reaching significance for the guidance of dense crowds and accident prevention, such as crowding and stampedes. As a typical multimodal scene understanding task, image captioning has always attracted widespread attention. However, crowd scene understanding captioning is rarely studied due to the unobtainability of related datasets. Therefore, it is difficult to know what happens in crowd scenes. In order to fill this research gap, we propose a crowd scenes caption dataset named CrowdCaption which has the advantages of crowd-topic scenes, comprehensive and complex caption descriptions, typical relationships and detailed grounding annotations. The complexity and diversity of the descriptions and the specificity of the crowd scenes make this dataset extremely challenging to most current methods. Thus, we propose a Multi-hierarchical Attribute Guided Crowd Caption Network (MAGC) based on crowd objects, actions, and status (such as position, dress, posture, etc.) aiming to generate crowd-specific detailed descriptions. We conduct extensive experiments on our CrowdCaption dataset, and our proposed method reaches the state-of-the-art (SoTA) performance. We hope the CrowdCaption dataset can assist future studies related to crowd scenes in the multimodal domain. Our dataset and the code of the benchmark method will be released at “https://ivipclab.github.io”.
- 20 July 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 25 (15209210) , 5400-5412
What Happens in Crowd Scenes: A New Dataset About Crowd Scenes for Image Captioning
Making machines endowed with eyes and brains to effectively understand and analyze crowd scenes is of paramount importance for building a smart city to serve people. This is of far-reaching significance for the guidance of dense crowds and accident prevention, such as crowding and stampedes. As a typical multimodal scene understanding task, image captioning has always attracted widespread attention. However, crowd scene understanding captioning is rarely studied due to the unobtainability of related datasets. Therefore, it is difficult to know what happens in crowd scenes. In order to fill this research gap, we propose a crowd scenes caption dataset named CrowdCaption which has the advantages of crowd-topic scenes, comprehensive and complex caption descriptions, typical relationships and detailed grounding annotations. The complexity and diversity of the descriptions and the specificity of the crowd scenes make this dataset extremely challenging to most current methods. Thus, we propose a Multi-hierarchical Attribute Guided Crowd Caption Network (MAGC) based on crowd objects, actions, and status (such as position, dress, posture, etc.) aiming to generate crowd-specific detailed descriptions. We conduct extensive experiments on our CrowdCaption dataset, and our proposed method reaches the state-of-the-art (SoTA) performance. We hope the CrowdCaption dataset can assist future studies related to crowd scenes in the multimodal domain. Our dataset and the code of the benchmark method will be released at “https://ivipclab.github.io”.
- 2 March 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 32 (2) , 704-714
Segmenting Beyond the Bounding Box for Instance Segmentation
Instance segmentation needs to locate all instances in an image correctly and segment each instance precisely. Currently, the most dominant methods for instance segmentation take object detection as a pre-task. However, they rely on the accuracy of object detection incredibly. If the pre-task cannot predict an accurate bounding box, the performance of instance segmentation will degenerate. In this paper, we present a novel method for instance segmentation to solve this problem, which is called S egmenting B eyond the B ounding B ox ( S3B-Net ). Our S3B-Net designs a sub-network to help instance segmentation methods based on object detection to segment the part of an instance beyond the bounding box. Specifically, the sub-network first predicts a two-dimensional pixel embedding for each pixel. Then, the Gaussian function is employed to calculate a pixel’s probability belongs to a corresponding instance according to the two-dimensional pixel embedding. Finally, the output of the sub-network combines with the output of instance segmentation based on object detection to generate a more precise instance mask. Our sub-network can easily extend on the existing instance segmentation method based on object detection to segment instance beyond the bounding box. We do our experiments on dominant instance segmentation datasets, such as the COCO dataset and Cityscapes dataset. The results show that our method can achieve 6.8 points gain compared with the baseline Mask R-CNN with ResNet-50-FPN in Cityscapes datasets, and 1.7 points gain with ResNet-101-FPN-DCN in COCO datasets. Our S3B-Net outperforms the previous state-of-the-art instance segmentation method, which proves our method is competitive. The source code of our method will be made available.
- 2 March 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 32 (2) , 704-714
Segmenting Beyond the Bounding Box for Instance Segmentation
Instance segmentation needs to locate all instances in an image correctly and segment each instance precisely. Currently, the most dominant methods for instance segmentation take object detection as a pre-task. However, they rely on the accuracy of object detection incredibly. If the pre-task cannot predict an accurate bounding box, the performance of instance segmentation will degenerate. In this paper, we present a novel method for instance segmentation to solve this problem, which is called S egmenting B eyond the B ounding B ox ( S3B-Net ). Our S3B-Net designs a sub-network to help instance segmentation methods based on object detection to segment the part of an instance beyond the bounding box. Specifically, the sub-network first predicts a two-dimensional pixel embedding for each pixel. Then, the Gaussian function is employed to calculate a pixel’s probability belongs to a corresponding instance according to the two-dimensional pixel embedding. Finally, the output of the sub-network combines with the output of instance segmentation based on object detection to generate a more precise instance mask. Our sub-network can easily extend on the existing instance segmentation method based on object detection to segment instance beyond the bounding box. We do our experiments on dominant instance segmentation datasets, such as the COCO dataset and Cityscapes dataset. The results show that our method can achieve 6.8 points gain compared with the baseline Mask R-CNN with ResNet-50-FPN in Cityscapes datasets, and 1.7 points gain with ResNet-101-FPN-DCN in COCO datasets. Our S3B-Net outperforms the previous state-of-the-art instance segmentation method, which proves our method is competitive. The source code of our method will be made available.
- Open Access
- 28 October 2020
- Published by MDPI AG in Remote Sensing
- Vol. 12 (21) , 3527
FDBP-InSAR: An Efficient Algorithm for InSAR Imaging via Frequency Domain Back Projection
High-quality focusing with accurate phase-preserving is a significant and challenging step in interferometric synthetic aperture radar (InSAR) imaging. Compared with conventional frequency-based imaging algorithms, the time-domain back-projection algorithm (TDBPA) can greatly ensure the accuracy of imaging and phase-preserving by point-to-point coherent integration but suffers from huge computational complexity. In this paper, we propose an efficient InSAR imaging method, called a frequency-domain back-projection algorithm (FDBPA), to achieve high-resolution focusing and accurate phase-preserving of InSAR imaging. More specifically, FDBPA is utilized to replace the traditional point-to-point coherent integration of TDBPA with frequency-domain transform. It divides the echo spectrum into uniform grids and transforms the range compression data into the range frequency domain. Phase compensation and non-uniform Fourier transform of the underlying scene are implemented to achieve image focusing in the wavenumber domain. Then, the interferometric phase of the target scene can be preserved by accurate phase compensation of the target’s distance. FDBPA avoids the repetitive calculation of index values and point-to-point coherent integration which reduces the time complexity compared with TDBPA. The characteristics of focusing and phase-preserving of our method are analyzed via simulations and experiments. The results demonstrate the efficiency and high-quality imaging of the FDBPA method. It can improve the imaging efficiency by more than three times, while keeping similar imaging accuracy compared with TDBPA.
- 28 October 2020
- Published by MDPI AG in Remote Sensing
- Vol. 12 (21) , 3527
Open AccessFDBP-InSAR: An Efficient Algorithm for InSAR Imaging via Frequency Domain Back Projection
High-quality focusing with accurate phase-preserving is a significant and challenging step in interferometric synthetic aperture radar (InSAR) imaging. Compared with conventional frequency-based imaging algorithms, the time-domain back-projection algorithm (TDBPA) can greatly ensure the accuracy of imaging and phase-preserving by point-to-point coherent integration but suffers from huge computational complexity. In this paper, we propose an efficient InSAR imaging method, called a frequency-domain back-projection algorithm (FDBPA), to achieve high-resolution focusing and accurate phase-preserving of InSAR imaging. More specifically, FDBPA is utilized to replace the traditional point-to-point coherent integration of TDBPA with frequency-domain transform. It divides the echo spectrum into uniform grids and transforms the range compression data into the range frequency domain. Phase compensation and non-uniform Fourier transform of the underlying scene are implemented to achieve image focusing in the wavenumber domain. Then, the interferometric phase of the target scene can be preserved by accurate phase compensation of the target’s distance. FDBPA avoids the repetitive calculation of index values and point-to-point coherent integration which reduces the time complexity compared with TDBPA. The characteristics of focusing and phase-preserving of our method are analyzed via simulations and experiments. The results demonstrate the efficiency and high-quality imaging of the FDBPA method. It can improve the imaging efficiency by more than three times, while keeping similar imaging accuracy compared with TDBPA.