Guangliang Zhou
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Guangliang Zhou has a total of 15 co-authors. Here are the top co-authors by number of shared publications:
- Qijun Chen
Tongji University
7 shared publications - Deming Wang
College of Electronics and Information Engineering, Tongji University, Shanghai, China
5 shared publications - Yi Yan
College of Electronics and Information Engineering, Tongji University, Shanghai, China
5 shared publications - Chengju Liu
Tongji University
2 shared publications - Feng Lin
Department of Electrical and Computer Engineering, Wayne State University, Detroit, MI, USA
1 shared publication Huiyi Chen
Tisch School of the Arts, New York University, New York, NY, USA
1 shared publication
Latest Publications
- 8 May 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 34 (10) , 9125-9138
CLIPose: Category-Level Object Pose Estimation With Pre-Trained Vision-Language Knowledge
Most of existing category-level object pose estimation methods devote to learning the object category information from point cloud modality. However, the scale of 3D datasets is limited due to the high cost of 3D data collection and annotation. Consequently, the category features extracted from these limited point cloud samples may not be comprehensive. This motivates us to investigate whether we can draw on knowledge of other modalities to obtain category information. Inspired by this motivation, we propose CLIPose, a novel 6D pose framework that employs the pre-trained vision-language model to develop better learning of object category information, which can fully leverage abundant semantic knowledge in image and text modalities. To make the 3D encoder learn category-specific features more efficiently, we align representations of three modalities in feature space via multi-modal contrastive learning. In addition to exploiting the pre-trained knowledge of the CLIP’s model, we also expect it to be more sensitive with pose parameters. Therefore, we introduce a prompt tuning approach to fine-tune image encoder while we incorporate rotations and translations information in the text descriptions. CLIPose achieves state-of-the-art performance on two mainstream benchmark datasets, REAL275 and CAMERA25, and runs in real-time during inference (40FPS).
- 8 May 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 34 (10) , 9125-9138
CLIPose: Category-Level Object Pose Estimation With Pre-Trained Vision-Language Knowledge
Most of existing category-level object pose estimation methods devote to learning the object category information from point cloud modality. However, the scale of 3D datasets is limited due to the high cost of 3D data collection and annotation. Consequently, the category features extracted from these limited point cloud samples may not be comprehensive. This motivates us to investigate whether we can draw on knowledge of other modalities to obtain category information. Inspired by this motivation, we propose CLIPose, a novel 6D pose framework that employs the pre-trained vision-language model to develop better learning of object category information, which can fully leverage abundant semantic knowledge in image and text modalities. To make the 3D encoder learn category-specific features more efficiently, we align representations of three modalities in feature space via multi-modal contrastive learning. In addition to exploiting the pre-trained knowledge of the CLIP’s model, we also expect it to be more sensitive with pose parameters. Therefore, we introduce a prompt tuning approach to fine-tune image encoder while we incorporate rotations and translations information in the text descriptions. CLIPose achieves state-of-the-art performance on two mainstream benchmark datasets, REAL275 and CAMERA25, and runs in real-time during inference (40FPS).
- 17 April 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Vehicles
- Vol. 9 (7) , 5163-5172
RoadFormer: Duplex Transformer for RGB-Normal Semantic Road Scene Parsing
The recent advancements in deep convolutional neural networks have shown significant promise in the domain of road scene parsing. Nevertheless, the existing works focus primarily on freespace detection, with little attention given to hazardous road defects that could compromise both driving safety and comfort. In this article, we introduce RoadFormer, a novel Transformer-based data-fusion network developed for road scene parsing. RoadFormer utilizes a duplex encoder architecture to extract heterogeneous features from both RGB images and surface normal information. The encoded features are subsequently fed into a novel heterogeneous feature synergy block for effective feature fusion and recalibration. The pixel decoder then learns multi-scale long-range dependencies from the fused and recalibrated heterogeneous features, which are subsequently processed by a Transformer decoder to produce the final semantic prediction. Additionally, we release SYN-UDTIRI, the first large-scale road scene parsing dataset that contains over 10,407 RGB images, dense depth images, and the corresponding pixel-level annotations for both freespace and road defects of different shapes and sizes. Extensive experimental evaluations conducted on our SYN-UDTIRI dataset, as well as on three public datasets, including KITTI road, CityScapes, and ORFD, demonstrate that RoadFormer outperforms all other state-of-the-art networks for road scene parsing. Specifically, RoadFormer ranks first on the KITTI road benchmark. Our source code, created dataset, and demo video are publicly available at mias.group/RoadFormer .
- 17 April 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Vehicles
- Vol. 9 (7) , 5163-5172
RoadFormer: Duplex Transformer for RGB-Normal Semantic Road Scene Parsing
The recent advancements in deep convolutional neural networks have shown significant promise in the domain of road scene parsing. Nevertheless, the existing works focus primarily on freespace detection, with little attention given to hazardous road defects that could compromise both driving safety and comfort. In this article, we introduce RoadFormer, a novel Transformer-based data-fusion network developed for road scene parsing. RoadFormer utilizes a duplex encoder architecture to extract heterogeneous features from both RGB images and surface normal information. The encoded features are subsequently fed into a novel heterogeneous feature synergy block for effective feature fusion and recalibration. The pixel decoder then learns multi-scale long-range dependencies from the fused and recalibrated heterogeneous features, which are subsequently processed by a Transformer decoder to produce the final semantic prediction. Additionally, we release SYN-UDTIRI, the first large-scale road scene parsing dataset that contains over 10,407 RGB images, dense depth images, and the corresponding pixel-level annotations for both freespace and road defects of different shapes and sizes. Extensive experimental evaluations conducted on our SYN-UDTIRI dataset, as well as on three public datasets, including KITTI road, CityScapes, and ORFD, demonstrate that RoadFormer outperforms all other state-of-the-art networks for road scene parsing. Specifically, RoadFormer ranks first on the KITTI road benchmark. Our source code, created dataset, and demo video are publicly available at mias.group/RoadFormer .
- 23 November 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Instrumentation and Measurement
- Vol. 72 (00189456) , 1-11
6-D Object Pose Estimation Using Multiscale Point Cloud Transformer
Predicting 6D object pose is an essential task in vision measurement for robotic manipulation. RGB-based methods have a natural disadvantage due to the lack of 3D information, thus leading to inferior results. Therefore, exploiting the geometry information in depth images is crucial to achieve accurate predictions. To this end, we propose a multi-scale point cloud Transformer (MSPCT) to better learn point cloud feature representations. MSPCT mainly consists of three types of modules: Local Transformer (LT), DownSampling module and Global Transformer (GT). Specifically, the Local Transformer is designed to dynamically divide a local region centered on each point and further extract point-level feature with local context awareness. The DownSampling module is utilized to decrease the resolution and enlarge the receptive field. Global Transformer is employed to capture global-range dependencies between extracted local features. Based on the proposed Transformer blocks, we design a network architecture for object pose estimation, where we further obtain multi-scale features by fusing the local features from LT and global features from GT to predict object’s pose. Extensive experiments verify the effectiveness of Local Transformer and Global Transformer, and our pose estimation pipeline achieves promising results on three benchmark datasets.
- 23 November 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Instrumentation and Measurement
- Vol. 72 (00189456) , 1-11
6-D Object Pose Estimation Using Multiscale Point Cloud Transformer
Predicting 6D object pose is an essential task in vision measurement for robotic manipulation. RGB-based methods have a natural disadvantage due to the lack of 3D information, thus leading to inferior results. Therefore, exploiting the geometry information in depth images is crucial to achieve accurate predictions. To this end, we propose a multi-scale point cloud Transformer (MSPCT) to better learn point cloud feature representations. MSPCT mainly consists of three types of modules: Local Transformer (LT), DownSampling module and Global Transformer (GT). Specifically, the Local Transformer is designed to dynamically divide a local region centered on each point and further extract point-level feature with local context awareness. The DownSampling module is utilized to decrease the resolution and enlarge the receptive field. Global Transformer is employed to capture global-range dependencies between extracted local features. Based on the proposed Transformer blocks, we design a network architecture for object pose estimation, where we further obtain multi-scale features by fusing the local features from LT and global features from GT to predict object’s pose. Extensive experiments verify the effectiveness of Local Transformer and Global Transformer, and our pose estimation pipeline achieves promising results on three benchmark datasets.
- 23 December 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 32 (8) , 5163-5174
Semi-Supervised 6D Object Pose Estimation Without Using Real Annotations
6D object pose estimation is a longstanding computer vision problem. Existing deep learning-based methods have achieved inspiring results in this task. However, large-scale training data with annotations is extremely needed to guarantee these methods’ performance, and acquiring real 6D object pose annotations is fairly labor-intensive and time-consuming. To overcome this drawback, we propose a semi-supervised pose estimation method using labeled synthetic data and unlabeled real data. For unlabeled real data, we form a self-supervised pipeline by minimizing the distance between the input point cloud, which is under ground-truth pose, and the model points transformed based on predicted pose. The labeled synthetic data is used to supervise the network to converge correctly. And we utilize a feature mapping to eliminate the domain gap between the real and synthetic features to further enhance the network’s performance. Moreover, we propose an attention-based pose estimation network, which can concentrate more on the distinguishing features, thus improving the accuracy of pose estimation. Experiments show that our proposed semi-supervised method is able to achieve good performance without the real annotations and outperforms all other methods relying on synthetic data or self-supervision strategy, indicating that the proposed method is effective.
- 23 December 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 32 (8) , 5163-5174
Semi-Supervised 6D Object Pose Estimation Without Using Real Annotations
6D object pose estimation is a longstanding computer vision problem. Existing deep learning-based methods have achieved inspiring results in this task. However, large-scale training data with annotations is extremely needed to guarantee these methods’ performance, and acquiring real 6D object pose annotations is fairly labor-intensive and time-consuming. To overcome this drawback, we propose a semi-supervised pose estimation method using labeled synthetic data and unlabeled real data. For unlabeled real data, we form a self-supervised pipeline by minimizing the distance between the input point cloud, which is under ground-truth pose, and the model points transformed based on predicted pose. The labeled synthetic data is used to supervise the network to converge correctly. And we utilize a feature mapping to eliminate the domain gap between the real and synthetic features to further enhance the network’s performance. Moreover, we propose an attention-based pose estimation network, which can concentrate more on the distinguishing features, thus improving the accuracy of pose estimation. Experiments show that our proposed semi-supervised method is able to achieve good performance without the real annotations and outperforms all other methods relying on synthetic data or self-supervision strategy, indicating that the proposed method is effective.
- 8 October 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 24 (15209210) , 4394-4408
GeoPose: Dense Reconstruction Guided 6D Object Pose Estimation With Geometric Consistency
6D object pose estimation for texture-less objects from RGB images remains challenging, especially in occlusion scenarios. Instead of localizing sparse keypoints by regressing their image coordinates or heatmaps, which are sensitive to occlusion, we introduce GeoPose, a novel reconstruction guided pose estimation pipeline that predicts dense correspondences and leverages geometric consistency effectively. We first design a dense reconstruction network (ReconNet) to reconstruct pixel-wise object coordinates in normalization space. Dense 2D-3D correspondences are generated intuitively by our explicit pa-rameterization for 3D object models, which dismisses keypoint selection efforts. These 2D-3D correspondences are then utilized to estimate 6D poses by the PnP algorithm with RANSAC iterations. Furthermore, a novel Cycle Loss is proposed to provide 3D prior supervision, which significantly correlates with the pose estimation task by guiding geometric consistency between reconstruction (pixel to 3D) and reprojection (3D to pixel). In addition, a training data augmentation method is proposed to handle the insufficiency of 6D datasets, the acquisition of which is error-prone and time-consuming. Extensive experiments demonstrate that, compared with existing RGB-based methods, our GeoPose can achieve state-of-the-art (SOTA) 6D pose estimation performance on the LINEMOD, Occlusion LINEMOD and T-LESS datasets.
- 8 October 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 24 (15209210) , 4394-4408
GeoPose: Dense Reconstruction Guided 6D Object Pose Estimation With Geometric Consistency
6D object pose estimation for texture-less objects from RGB images remains challenging, especially in occlusion scenarios. Instead of localizing sparse keypoints by regressing their image coordinates or heatmaps, which are sensitive to occlusion, we introduce GeoPose, a novel reconstruction guided pose estimation pipeline that predicts dense correspondences and leverages geometric consistency effectively. We first design a dense reconstruction network (ReconNet) to reconstruct pixel-wise object coordinates in normalization space. Dense 2D-3D correspondences are generated intuitively by our explicit pa-rameterization for 3D object models, which dismisses keypoint selection efforts. These 2D-3D correspondences are then utilized to estimate 6D poses by the PnP algorithm with RANSAC iterations. Furthermore, a novel Cycle Loss is proposed to provide 3D prior supervision, which significantly correlates with the pose estimation task by guiding geometric consistency between reconstruction (pixel to 3D) and reprojection (3D to pixel). In addition, a training data augmentation method is proposed to handle the insufficiency of 6D datasets, the acquisition of which is error-prone and time-consuming. Extensive experiments demonstrate that, compared with existing RGB-based methods, our GeoPose can achieve state-of-the-art (SOTA) 6D pose estimation performance on the LINEMOD, Occlusion LINEMOD and T-LESS datasets.
- 24 August 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Instrumentation and Measurement
- Vol. 70 (00189456) , 1-12
Unsupervised Anomaly Segmentation Via Multilevel Image Reconstruction and Adaptive Attention-Level Transition
In product inspection scenarios, the objects have different characteristics of textures and different potential anomalies. Still, traditional methods cannot precisely segment structural anomalies and achieve optimal performance on all object categories. This article introduces a flexible multilevel image reconstruction (MLIR) framework for the unsupervised anomaly segmentation task. The MLIR integrates multiple generators with different downsampling ratios to serve as multiple reconstruction levels and leverage an adjustable perceptual similarity measurement to obtain the featurewise reconstruction errors, overcoming the challenge of detecting multiscale anomalies. Leveraging embeddings with different receptive fields for reconstruction, generators perform differently on the same target. Therefore, we depend on the reconstruction uncertainty to propose an adaptive attention-level transition (ALT) strategy that optimizes the detection performance automatically according to the target's category. ALT adjusts the weights of both reconstruction levels and feature measurement scales simultaneously to utilize features at a consistent level for reconstruction and anomaly detection (AD). This strategy remarkably improves the system's adaptivity for different applications. Moreover, we propose reducing the signal-to-noise ratio (SNR) of the images by adding pepper noise so that the reconstruction process is framed as an image denoising task, which improves feature extraction and image restoration robustness. We conduct extensive experiments on the MVTec AD dataset and demonstrate that the proposed method performs satisfactorily to different categories and achieves the state-of-the-art (SOTA) performance.
- 24 August 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Instrumentation and Measurement
- Vol. 70 (00189456) , 1-12
Unsupervised Anomaly Segmentation Via Multilevel Image Reconstruction and Adaptive Attention-Level Transition
In product inspection scenarios, the objects have different characteristics of textures and different potential anomalies. Still, traditional methods cannot precisely segment structural anomalies and achieve optimal performance on all object categories. This article introduces a flexible multilevel image reconstruction (MLIR) framework for the unsupervised anomaly segmentation task. The MLIR integrates multiple generators with different downsampling ratios to serve as multiple reconstruction levels and leverage an adjustable perceptual similarity measurement to obtain the featurewise reconstruction errors, overcoming the challenge of detecting multiscale anomalies. Leveraging embeddings with different receptive fields for reconstruction, generators perform differently on the same target. Therefore, we depend on the reconstruction uncertainty to propose an adaptive attention-level transition (ALT) strategy that optimizes the detection performance automatically according to the target's category. ALT adjusts the weights of both reconstruction levels and feature measurement scales simultaneously to utilize features at a consistent level for reconstruction and anomaly detection (AD). This strategy remarkably improves the system's adaptivity for different applications. Moreover, we propose reducing the signal-to-noise ratio (SNR) of the images by adding pepper noise so that the reconstruction process is framed as an image denoising task, which improves feature extraction and image restoration robustness. We conduct extensive experiments on the MVTec AD dataset and demonstrate that the proposed method performs satisfactorily to different categories and achieves the state-of-the-art (SOTA) performance.
- 11 June 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 23 (15209210) , 1630-1639
A Novel Depth and Color Feature Fusion Framework for 6D Object Pose Estimation
This paper aims to solve the problem of estimating the 6D pose of an object under severe occlusion using RGB-D images. Most existing methods typically use the information of color and depth images separately to make predictions, which limits their performances in the presence of occlusion. Instead, we propose a pipeline to effectively fuse color and depth information and perform region-level pose estimation. Our method first uses a CNN to extract the color features, and then we obtain the fusion features by combining the color features into the point cloud. Unlike existing methods, the fusion features are in the form of point sets instead of feature maps. We further use a PointNet++-like network to process the fusion features, obtaining several region-level features. Each region-level feature can predict a pose with confidence. The pose with the highest confidence is chosen as the final output. Experiments show that the proposed method outperforms the-state-of-the-art methods on both the LINEMOD and Occlusion LINEMOD datasets, indicating that the proposed pipeline can obtain accurate pose estimation results and is robust to occlusion.
- 11 June 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 23 (15209210) , 1630-1639
A Novel Depth and Color Feature Fusion Framework for 6D Object Pose Estimation
This paper aims to solve the problem of estimating the 6D pose of an object under severe occlusion using RGB-D images. Most existing methods typically use the information of color and depth images separately to make predictions, which limits their performances in the presence of occlusion. Instead, we propose a pipeline to effectively fuse color and depth information and perform region-level pose estimation. Our method first uses a CNN to extract the color features, and then we obtain the fusion features by combining the color features into the point cloud. Unlike existing methods, the fusion features are in the form of point sets instead of feature maps. We further use a PointNet++-like network to process the fusion features, obtaining several region-level features. Each region-level feature can predict a pose with confidence. The pose with the highest confidence is chosen as the final output. Experiments show that the proposed method outperforms the-state-of-the-art methods on both the LINEMOD and Occlusion LINEMOD datasets, indicating that the proposed pipeline can obtain accurate pose estimation results and is robust to occlusion.