Zhiqiang Cao

Category-level 6D object pose estimation has gained increasing attention in applications of robotic manipulation, augmented reality, and scene understanding, due to its ability to generalize to unseen instances within the same category. However, existing methods struggle with handling the intra-class shape variations as they either adopt mean shape as priors, or build the associations among different instances without explicit category-shared information. To address this problem, a novel category-level object pose estimation method based on diffusion model is proposed, which utilize the generative ability of diffusion model to refine a sparse categorical representation. In contrast to existing dense correspondence-based methods, our method employs a set of keypoints provided by learnable queries to represent object shape, enabling better categorical representation of different instances by focusing on the representative object components. The keypoints are then refined through a forward diffusion process and a reverse denoising process conditioned on category information. This allows the flexible adaption to various instances, especially for those that deviate from the mean shape within the same category. On this basis, a geometric-semantic feature fusion module is presented to enhance keypoint feature representation. By integrating the geometric information from point cloud with the high-level semantics from RGB image using a two-branch attention mechanism, the keypoint feature is enriched and deeply combined, which facilitates the subsequent pose estimation. Extensive experiments on the REAL275 dataset, the CAMERA25 dataset, and real-world complex scenarios demonstrated the effectiveness of proposed method.

Few-shot object detection receives much attention with the ability to detect novel class objects using limited annotated data. The transfer learning-based solution becomes popular due to its simple training with good accuracy, however, it is still challenging to enrich the feature diversity during the training process. And fine-grained features are also insufficient for novel class detection. To deal with the problems, this paper proposes a novel few-shot object detection method based on dual-domain feature fusion and patch-level attention. Upon original base domain, an elementary domain with more category-agnostic features is superposed to construct a two-stream backbone, which benefits to enrich the feature diversity. To better integrate various features, a dual-domain feature fusion is designed, where the feature pairs with the same size are complementarily fused to extract more discriminative features. Moreover, a patch-wise feature refinement termed as patch-level attention is presented to mine internal relations among the patches, which enhances the adaptability to novel classes. In addition, a weighted classification loss is given to assist the fine-tuning of the classifier by combining extra features from FPN of the base training model. In this way, the few-shot detection quality to novel class objects is improved. Experiments on PASCAL VOC and MS COCO datasets verify the effectiveness of the method.

The next best view (NBV) plays an important role in the three-dimensional object reconstruction. Existing NBV methods mainly adopt the generate-and-test strategy to select the best view in the discrete view space. This affects the adaptation to diverse objects. To solve this problem, a new NBV paradigm to directly generate the next best view in the continuous view space according to the input point cloud is proposed. Specifically, a point cloud feature extraction module with learnable view and position tokens is presented. These tokens are added upon the neighborhood and the position features of the point cloud to fully mine global contextual information, enhancing the feature representation. The predicted view from the proposed network is linked to a pre-trained view evaluation network. By duplicating this prediction view and then concatenating the duplication result with the extracted point cloud feature, the evaluation network is endowed with the ability to evaluate arbitrary views. Take the evaluation score of the evaluation network corresponding to the predicted view as the supervision signal, the network is trained. In this way, an effective solution of the next best view selection in the continuous view space is obtained. Accordingly, the adaptability to different objects is reinforced. Experimental results on the ShapeNet and MIT CSAIL datasets demonstrate the effectiveness of the proposed method.

Autofocus is a crucial prerequisite for precision vision sensing of industrial workpieces. Existing methods mainly utilize image sharpness to guide single-axis camera adjustment for focusing. However, they fail to deal with larger workpieces with uneven surfaces, which require multidirection sensing. To solve this challenge, the focusing is abstracted as a force movement acting on the camera focus plane, and a defocus force-guided multi-DOF (degree of freedom) autofocus method with a hand–eye robot system is proposed. For better force guidance, the image is divided into blocks and a defocus degree from a defocus evaluation network is associated with each block. By taking advantage of the dilated convolution with an attention, the defocus evaluation network achieves high precision while satisfying lightweight characteristics. For each image block, its corresponding defocus force is constructed with the amplitude of its defocus degree. All block defocus forces are then synthesized to the resultant force and torque acting on the center of the focus plane, which are employed to drive the manipulator to adjust the position and posture of the camera, respectively. This promotes the high-precision multi-DOF autofocus on uneven surfaces. Experimental results on industrial workpieces demonstrate the effectiveness of the proposed method.

Robot grasp modeling and implementation is one of essential abilities for a robot with embodied artificial intelligence. However, most existing deep learning-based grasp methods suffer a large number of parameters and heavy computation overheads. To address this issue, by fully exploiting the complementary capabilities of both CNNs and Transformers, we propose a smart grasp deep network with one-way fusion strategy via context path and spatial path (SGNet), which enjoys a lightweight structure, fast inference speed and easy deployment on devices with limited computation resources. Specifically, the context path employs lightweight depthwise separable convolution to achieve fast down-sampling while a novel DSFormer module mainly by integrating Transformer is to extract global and context-rich features. The spatial path efficiently fuses feature information from the context path in one-way manner and generate high-resolution feature maps via point-by-point convolution operations. Experimental results show the proposed model with only 1M parameters has a significantly overall performance, achieving 99.4% accuracy on Cornell dataset and 93.4% accuracy on Jacquard dataset, as well as within 12.5ms inference time.

Visual place recognition is crucial to accurate localization in large-scale environments. Existing methods combine convolutional neural network and deep metric learning to improve performance, however, it is still challenging to promote the adaptability of features under various environmental conditions. To address the problem, this paper proposes a hardness-aware metric learning method with cluster-guided attention. By leveraging the affinity between each local feature and the corresponding scene cluster center, the model is attracted to focus on the local features proximate to their clusters, while suppressing outlier local features that deviate from their clusters. In this way, the scene-related reliable local features are concentrated to construct the global feature of the whole image with adaptability to environmental conditions. Meanwhile, a hardness-aware metric loss is designed to train the proposed network, which determines the hardness of negative samples based on their similarity to the query images and the training iterations. Subsequently, the hardness is used to reweigh each term of the loss function to promote network optimization. In addition, a condition normalization layer is also introduced to regularize the feature distributions under different environmental conditions to a canonical space, improving the feature robustness to condition variations. Our method achieves the top-10 recalls of 97.2%, 99.0%, and 94.9% on Pitts250k-test, TokyoTM-val, and Tokyo 24/7 datasets, respectively. Extensive experiments demonstrate that the proposed method learns robust global features with the adaptability to various environmental conditions.

Multisensor fusion simultaneous localization and mapping (SLAM) has gained popularity in the SLAM community due to its low cost and high real-time performance. Common point-feature-based visual-inertial SLAM systems often struggle in environments with weak textures or motion blur. By incorporating line features, the accuracy and robustness of SLAM systems can be improved. However, challenges in line matching and increased processing time caused by line features have limited these improvements. To address the problem, we introduce a real-time monocular visual-inertial SLAM method with IMU-aided hybrid line matching, where the hybrid lines consist of elementary and recessive lines. Specifically, an IMU-aided hybrid line matching scheme is designed to determine the search space of line matching according to the IMU pre-integration result. It scales down the search range effectively and thus improves the accuracy and speed of line matching. Also, an improved enhanced line segment drawing (ELSED) algorithm is utilized for efficient elementary line feature extraction, where the parameters of line features are adaptively adjusted with the number of extracted point features to avoid feature redundancy. In addition, we also extend the point-based loop-closure detection by introducing line features for higher accuracy of loop-closure detection. Experiment results demonstrate the effectiveness of the proposed method.