He Cao
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
He Cao has a total of 6 co-authors. Here are the top co-authors by number of shared publications:
- Yunzhou Zhang
College of Information Science and Engineering, Northeastern University, Shenyang, China
3 shared publications - Lei Wang
Faculty of Robot Science and Engineering, Northeastern University, Shenyang, China
2 shared publications - Shangdong Zhu
School of Communication Engineering, Hangzhou Dianzi University, Hangzhou, China
2 shared publications - Le Zhou
Faculty of Robot Science and Engineering, Northeastern University, Shenyang, China
1 shared publication Lei Wang
Northeastern Univ, Fac Robot Sci & Engn, Shenyang 110169, Peoples R China
1 shared publication- Yixiu Liu
Hangzhou Dianzi University
1 shared publication
Latest Publications
- 13 March 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Instrumentation and Measurement
- Vol. 74 (00189456) , 1-11
Improving Grasp Pose Detection by Implicitly Utilizing Geometric Information and Spatial Relations of Objects in Clutter
6-DoF grasp pose detection in clutter is an essential but challenging task in the vision measurement. However, most current detection methods often fail to consider the geometric information and partial occlusions of objects in clutter, resulting in inaccurate robotic grasping and potential collisions. In this work, we propose an end-to-end grasping system to directly generate 6-DoF grasp poses from point clouds in clutter. Our system first selects the most feasible positions as the grasp centers, and then determines their grasp parameters, such as approaching vectors and in-plane rotations. Specifically, we design an optimality criterion based on the geometric information of objects to extract rich object features from the scene and identify high-graspablity areas. To alleviate the collision problem caused by occlusions, we focus on the spatial relations among objects and explicitly train a collision predictor to directly select collision-free points as grasp centers from high-graspablity areas. Based on these grasp centers, our model can easily generate diverse sets of collision- free and robust grasp poses for successful grasping. To efficiently determine grasp parameters, we exploit neighbor search to capture the local context around each grasp center and then implicitly learn relations among objects using the self-attention mechanism. Compared with previous methods, our grasping system improves performance by a large margin (>10%), and achieves state-of-the-art results on GraspNet-1Billion, a largescale dataset. Besides, our method has a high success rate for grasping in simulation and real-world experiments, verifying the validity in clutter.
- 13 March 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Instrumentation and Measurement
- Vol. 74 (00189456) , 1-11
Improving Grasp Pose Detection by Implicitly Utilizing Geometric Information and Spatial Relations of Objects in Clutter
6-DoF grasp pose detection in clutter is an essential but challenging task in the vision measurement. However, most current detection methods often fail to consider the geometric information and partial occlusions of objects in clutter, resulting in inaccurate robotic grasping and potential collisions. In this work, we propose an end-to-end grasping system to directly generate 6-DoF grasp poses from point clouds in clutter. Our system first selects the most feasible positions as the grasp centers, and then determines their grasp parameters, such as approaching vectors and in-plane rotations. Specifically, we design an optimality criterion based on the geometric information of objects to extract rich object features from the scene and identify high-graspablity areas. To alleviate the collision problem caused by occlusions, we focus on the spatial relations among objects and explicitly train a collision predictor to directly select collision-free points as grasp centers from high-graspablity areas. Based on these grasp centers, our model can easily generate diverse sets of collision- free and robust grasp poses for successful grasping. To efficiently determine grasp parameters, we exploit neighbor search to capture the local context around each grasp center and then implicitly learn relations among objects using the self-attention mechanism. Compared with previous methods, our grasping system improves performance by a large margin (>10%), and achieves state-of-the-art results on GraspNet-1Billion, a largescale dataset. Besides, our method has a high success rate for grasping in simulation and real-world experiments, verifying the validity in clutter.
- 23 October 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Instrumentation and Measurement
- Vol. 73 (00189456) , 1-18
PRO-CLIP: A CLIP-Based Category Measurement Network Through Prototype and Regularized Optimal Transportation
In unstructured environments, robots are likely to encounter desktop objects that they have never seen before. Classifying these objects precisely is a prerequisite for accomplishing object-specific manipulation tasks. However, it is time-consuming to collect large-scale object classification datasets. Inspired by the prompt tuning methods, we propose the PRO-CLIP network, which is a category measurement method for desktop objects. Specifically, PRO-CLIP performs few-shot classification based on the knowledge from pretrained vision-language model. It utilizes token-level and prompt-level optimal transportations to jointly fine-tune the learnable vision-language prompts. For token-level stage, we propose the image patch reweighting mechanism to make alignments focus on the image patches that are close to the patch prototypes. This allows the patch embeddings have converging category representations, which reduces intra-class differences of visual features. For prompt-level stage, we propose a cascading optimal transportation module to simultaneously consider task-irrelevant knowledge in zero-shot features and task-specific knowledge in few-shot features. Due to the generalization performance of task-irrelevant knowledge, the proposed module achieves feature regularization during optimal transportation. Finally, we propose the UP loss to supervise the whole network. It contains unbalanced logit-level consistency losses and visual prototype loss. The logit-level consistency losses are used to make learnable features close to zero-shot features. The prototype loss makes the visual features approach to the corresponding prototypes in distance. We demonstrate the effectiveness of our method by performing few-shot classification experiments on different datasets including desktop objects. The relevant code will be available at https://github.com/NeuCV-IRMI/proclip.
- 23 October 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Instrumentation and Measurement
- Vol. 73 (00189456) , 1-18
PRO-CLIP: A CLIP-Based Category Measurement Network Through Prototype and Regularized Optimal Transportation
In unstructured environments, robots are likely to encounter desktop objects that they have never seen before. Classifying these objects precisely is a prerequisite for accomplishing object-specific manipulation tasks. However, it is time-consuming to collect large-scale object classification datasets. Inspired by the prompt tuning methods, we propose the PRO-CLIP network, which is a category measurement method for desktop objects. Specifically, PRO-CLIP performs few-shot classification based on the knowledge from pretrained vision-language model. It utilizes token-level and prompt-level optimal transportations to jointly fine-tune the learnable vision-language prompts. For token-level stage, we propose the image patch reweighting mechanism to make alignments focus on the image patches that are close to the patch prototypes. This allows the patch embeddings have converging category representations, which reduces intra-class differences of visual features. For prompt-level stage, we propose a cascading optimal transportation module to simultaneously consider task-irrelevant knowledge in zero-shot features and task-specific knowledge in few-shot features. Due to the generalization performance of task-irrelevant knowledge, the proposed module achieves feature regularization during optimal transportation. Finally, we propose the UP loss to supervise the whole network. It contains unbalanced logit-level consistency losses and visual prototype loss. The logit-level consistency losses are used to make learnable features close to zero-shot features. The prototype loss makes the visual features approach to the corresponding prototypes in distance. We demonstrate the effectiveness of our method by performing few-shot classification experiments on different datasets including desktop objects. The relevant code will be available at https://github.com/NeuCV-IRMI/proclip.
- 22 October 2024
- Published by IOP Publishing in Measurement Science and Technology
- Vol. 36 (1) , 015410
IVA-former: invisible–visible query guided amodal mask measurement network for desktop object via hierarchical transformer
Instance segmentation of desktop objects is important for service robots. Most of the previous works for desktop environments are restricted to measuring the visible area of target objects. However, when a target object is placed behind another, the algorithm that only performs visible area segmentation is unable to provide accurate appearance information for the occluded objects. To solve this problem, we propose the invisible–visible query guided amodal mask measurement network based on a hierarchical transformer for desktop scenes, which can perceive the entire appearance of objects in the presence of occlusions. In this method, the RGB-D backbone is adopted to fuse the features from both RGB and depth images. Then, the pixel decoder is used to generate multi-scale feature maps. We then adopt a hierarchical transformer decoder to predict invisible, visible, and amodal masks simultaneously. To enhance the associations between the three prediction branches, we propose a query transform module to transfer object queries between adjacent branches. Since amodal masks are a combination of invisible and visible masks, we propose an invisible–visible mixture loss that takes masks from both invisible and visible branches to further supervise the network. Our method is trained on synthetic datasets for desktop objects and evaluated on both visible and amodal real-world datasets. Compared to other recent segmentation algorithms, our method achieves competitive performance.
- 22 October 2024
- Published by IOP Publishing in Measurement Science and Technology
- Vol. 36 (1) , 015410
IVA-former: invisible–visible query guided amodal mask measurement network for desktop object via hierarchical transformer
Instance segmentation of desktop objects is important for service robots. Most of the previous works for desktop environments are restricted to measuring the visible area of target objects. However, when a target object is placed behind another, the algorithm that only performs visible area segmentation is unable to provide accurate appearance information for the occluded objects. To solve this problem, we propose the invisible–visible query guided amodal mask measurement network based on a hierarchical transformer for desktop scenes, which can perceive the entire appearance of objects in the presence of occlusions. In this method, the RGB-D backbone is adopted to fuse the features from both RGB and depth images. Then, the pixel decoder is used to generate multi-scale feature maps. We then adopt a hierarchical transformer decoder to predict invisible, visible, and amodal masks simultaneously. To enhance the associations between the three prediction branches, we propose a query transform module to transfer object queries between adjacent branches. Since amodal masks are a combination of invisible and visible masks, we propose an invisible–visible mixture loss that takes masks from both invisible and visible branches to further supervise the network. Our method is trained on synthetic datasets for desktop objects and evaluated on both visible and amodal real-world datasets. Compared to other recent segmentation algorithms, our method achieves competitive performance.