Zhiyuan Zhang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Zhiyuan Zhang has a total of 38 co-authors. Here are the top co-authors by number of shared publications:
- Junpeng Shi
National University of Defense Technology
7 shared publications - Yuchao Dai
School of Electronics and Information, Northwestern Polytechnical University, Xian, China
5 shared publications - Bin Fan
School of Electronics and Information, Northwestern Polytechnical University, Xian, China
4 shared publications - Mingyi He
School of Electronics and Information, Northwestern Polytechnical University, Xian, China
4 shared publications - Jiadai Sun
Northwestern Polytechnical University
3 shared publications - Yongxiang Liu
National University of Defense Technology
3 shared publications
Latest Publications
- Open Access
- 16 April 2026
- Published by MDPI AG in Remote Sensing
- Vol. 18 (8) , 1196
Deep Learning Methods for SAR and Optical Image Fusion: A Review
Synthetic Aperture Radar (SAR) and optical image fusion technology plays a crucial role in remote sensing applications. It effectively combines the high spatial resolution and rich spectral information of optical images with the all-weather and penetrating observation advantages of SAR images, thereby significantly enhancing image interpretation accuracy and task execution capabilities. This paper systematically reviews deep learning-based fusion methods for SAR and optical images, with a particular focus on recent advances in deep learning models. Furthermore, it summarizes commonly used evaluation metrics for assessing fusion image quality, providing a basis for comparing and analyzing the performance of different methods. In addition, commonly used SAR-optical fusion datasets are briefly reviewed to highlight their roles in algorithm development and performance evaluation. Unlike conventional review articles, this paper further analyzes the guidance and supporting role of fusion algorithms from the perspective of typical and specific applications. Finally, it identifies key challenges and issues faced by current fusion methods, including data registration, model lightweight design, and multimodal feature alignment, and offers perspectives on future research directions. This review aims to provide routes and references for the development of SAR and optical image fusion technology.
- 16 April 2026
- Published by MDPI AG in Remote Sensing
- Vol. 18 (8) , 1196
Open AccessDeep Learning Methods for SAR and Optical Image Fusion: A Review
Synthetic Aperture Radar (SAR) and optical image fusion technology plays a crucial role in remote sensing applications. It effectively combines the high spatial resolution and rich spectral information of optical images with the all-weather and penetrating observation advantages of SAR images, thereby significantly enhancing image interpretation accuracy and task execution capabilities. This paper systematically reviews deep learning-based fusion methods for SAR and optical images, with a particular focus on recent advances in deep learning models. Furthermore, it summarizes commonly used evaluation metrics for assessing fusion image quality, providing a basis for comparing and analyzing the performance of different methods. In addition, commonly used SAR-optical fusion datasets are briefly reviewed to highlight their roles in algorithm development and performance evaluation. Unlike conventional review articles, this paper further analyzes the guidance and supporting role of fusion algorithms from the perspective of typical and specific applications. Finally, it identifies key challenges and issues faced by current fusion methods, including data registration, model lightweight design, and multimodal feature alignment, and offers perspectives on future research directions. This review aims to provide routes and references for the development of SAR and optical image fusion technology.
- 12 March 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 35 (10577149) , 2625-2639
Detection Drives an End-to-End Fusion of Infrared and Visible Images Based on Diffusion Models
Infrared and visible image fusion methods have shown promising results, yet existing approaches either compromise downstream detection performance through independent fusion processes or sacrifice computational efficiency and flexibility by requiring joint training of fusion and detection models. To address these challenges, we propose a detection-driven image fusion network based on diffusion models (termed as DDIF), which optimizes the fused images specifically for object detection tasks. Our method features the following three aspects: 1) We reformulate the image fusion process as an inverse problem solved by a non-differentiable optimization process wherein the fused result preserves the source modality information while conforming to the image prior provided by the diffusion model. 2) We design a Response Guide Learning Module (RGLM) to learn response maps, which determine the contribution of each modality in the fusion process according to the downstream detection task. 3) We establish explicit gradient relationships to ensure compatibility between RGLM training and the non-differentiable optimization process, enabling end-to-end training. Notably, a moderate coupling mechanism is formed in our frame-work as the subsequent detection model is pre-trained and frozen, enabling flexible integration with various advanced detection networks while maintaining computational efficiency. Extensive experiments indicate that our method achieves superior detection performance compared to SOTA approaches and produces high-quality image fusion results.
- 12 March 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 35 (10577149) , 2625-2639
Detection Drives an End-to-End Fusion of Infrared and Visible Images Based on Diffusion Models
Infrared and visible image fusion methods have shown promising results, yet existing approaches either compromise downstream detection performance through independent fusion processes or sacrifice computational efficiency and flexibility by requiring joint training of fusion and detection models. To address these challenges, we propose a detection-driven image fusion network based on diffusion models (termed as DDIF), which optimizes the fused images specifically for object detection tasks. Our method features the following three aspects: 1) We reformulate the image fusion process as an inverse problem solved by a non-differentiable optimization process wherein the fused result preserves the source modality information while conforming to the image prior provided by the diffusion model. 2) We design a Response Guide Learning Module (RGLM) to learn response maps, which determine the contribution of each modality in the fusion process according to the downstream detection task. 3) We establish explicit gradient relationships to ensure compatibility between RGLM training and the non-differentiable optimization process, enabling end-to-end training. Notably, a moderate coupling mechanism is formed in our frame-work as the subsequent detection model is pre-trained and frozen, enabling flexible integration with various advanced detection networks while maintaining computational efficiency. Extensive experiments indicate that our method achieves superior detection performance compared to SOTA approaches and produces high-quality image fusion results.
- 27 November 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Aerospace and Electronic Systems
- Vol. 62 (00189251) , 2403-2418
Adversarial Noise-Robust Modulation Recognition via Contrastive Latent Space Regularization
Automatic modulation classification (AMC) enables automated wireless spectrum anomaly detection, facilitating efficient spectrum enforcement. Deep neural networks (DNNs), characterized by multiple hidden layers with nonlinear activation functions that allow for learning higher-level patterns hidden in the data, have proven effective in various tasks, including AMC. However, existing DNN-based approaches often struggle with generalization—their performance degrades significantly when deployed under unseen conditions, such as varying signal-to-noise ratios (SNRs) distinct from their training environments. To address this limitation, we propose CTLAE, a supervised deep learning framework that integrates an auto-encoder architecture with contrastive learning. During training, CTLAE processes signal samples of the same modulation category but with diverse SNRs, explicitly guiding the model to discern noise-intensity discrepancies. Additionally, the model is tasked with reconstructing segmented and heavily masked signal data, further enhancing its robustness to noise variations. Experiments on radar and communication signal datasets validate CTLAE's superiority: it achieves at least 5% higher overall test accuracy compared to state-of-the-art baselines, demonstrating significant improvements in generalization.
- 27 November 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Aerospace and Electronic Systems
- Vol. 62 (00189251) , 2403-2418
Adversarial Noise-Robust Modulation Recognition via Contrastive Latent Space Regularization
Automatic modulation classification (AMC) enables automated wireless spectrum anomaly detection, facilitating efficient spectrum enforcement. Deep neural networks (DNNs), characterized by multiple hidden layers with nonlinear activation functions that allow for learning higher-level patterns hidden in the data, have proven effective in various tasks, including AMC. However, existing DNN-based approaches often struggle with generalization—their performance degrades significantly when deployed under unseen conditions, such as varying signal-to-noise ratios (SNRs) distinct from their training environments. To address this limitation, we propose CTLAE, a supervised deep learning framework that integrates an auto-encoder architecture with contrastive learning. During training, CTLAE processes signal samples of the same modulation category but with diverse SNRs, explicitly guiding the model to discern noise-intensity discrepancies. Additionally, the model is tasked with reconstructing segmented and heavily masked signal data, further enhancing its robustness to noise variations. Experiments on radar and communication signal datasets validate CTLAE's superiority: it achieves at least 5% higher overall test accuracy compared to state-of-the-art baselines, demonstrating significant improvements in generalization.
- 17 November 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Instrumentation and Measurement
- Vol. 74 (00189456) , 1-15
IDE: Illumination Disentanglement by an Equivariant Network for Dark Visible and Infrared Image Fusion
As a crucial image processing technology in response to the limited capture of spectral ranges and information of visible and infrared cameras, infrared and visible image fusion is devoted to producing high-quality fused measurement with the advantages of source images, which has drawn widespread attention and encouraged various applications. However, the existing methods are usually designed for images in good condition, while the extreme illumination environment will bring huge challenges due to the weak texture details and poor visual perception presented by the visible images. To this end, this paper proposes a new method (termed as IDE) for dark visible and infrared image fusion from the perspective of illumination disentanglement by extending the equivariance property. Specifically, our IDE is a two-stage method including an illumination disentanglement stage (ID-Stage) and an attributes enhancement stage (AE-Stage). ID-Stage employs a Siamese structure with two shared branches. The input image pair for each branch is augmented by diverse illuminations, where the extended equivariance property constructs an inherent self-supervised constraint on two branch outputs. This constraint is used to facilitate the training in addition to the image reconstruction loss to decouple the illumination disturbance explicitly. The AE-stage is devoted to retaining enough information from the original input and augmenting the crucial attributes for learning the fused image. Thereby, the key attributes, including the gradient, density, and contrast, are enhanced further by multiple attention blocks. In this way, the bright fused image with the salient object, meticulously texture, and detailed background is achieved. Extensive experiments are conducted to evaluate our IDE. The superior performance in terms of visual quality and quantitative evaluations demonstrates that our method achieves new state-of-the-art (SOTA) results towards fusing the dark visible image and infrared image.
- 17 November 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Instrumentation and Measurement
- Vol. 74 (00189456) , 1-15
IDE: Illumination Disentanglement by an Equivariant Network for Dark Visible and Infrared Image Fusion
As a crucial image processing technology in response to the limited capture of spectral ranges and information of visible and infrared cameras, infrared and visible image fusion is devoted to producing high-quality fused measurement with the advantages of source images, which has drawn widespread attention and encouraged various applications. However, the existing methods are usually designed for images in good condition, while the extreme illumination environment will bring huge challenges due to the weak texture details and poor visual perception presented by the visible images. To this end, this paper proposes a new method (termed as IDE) for dark visible and infrared image fusion from the perspective of illumination disentanglement by extending the equivariance property. Specifically, our IDE is a two-stage method including an illumination disentanglement stage (ID-Stage) and an attributes enhancement stage (AE-Stage). ID-Stage employs a Siamese structure with two shared branches. The input image pair for each branch is augmented by diverse illuminations, where the extended equivariance property constructs an inherent self-supervised constraint on two branch outputs. This constraint is used to facilitate the training in addition to the image reconstruction loss to decouple the illumination disturbance explicitly. The AE-stage is devoted to retaining enough information from the original input and augmenting the crucial attributes for learning the fused image. Thereby, the key attributes, including the gradient, density, and contrast, are enhanced further by multiple attention blocks. In this way, the bright fused image with the salient object, meticulously texture, and detailed background is achieved. Extensive experiments are conducted to evaluate our IDE. The superior performance in terms of visual quality and quantitative evaluations demonstrates that our method achieves new state-of-the-art (SOTA) results towards fusing the dark visible image and infrared image.
- 2 July 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Aerospace and Electronic Systems
- Vol. PP (00189251) , 1-15
D $^{2}$ AF-Net: A Dual-Domain Adaptive Fusion Method for Radar Deception Jamming RecognitionDeception jamming has garnered widespread attention for threatening radar systems due to its lower energy requirement compared to traditional suppression jamming. With the rapid evolution of electronic warfare technologies, multiple-input multiple-output (MIMO) radar has demonstrated remarkable progress in countering jamming. Non-networked deception jammers, capable of generating compound deception jamming within MIMO radar's main lobe, present significant challenges. However, existing jamming recognition methods suffer from static assumptions tailored to limited jamming patterns, excessive dependence on prior knowledge, inefficacy in extracting discriminative features, and high computational cost. To overcome these limitations, we first propose the dual-domain adaptive fusion network (D $^{2}$ AF-Net). This innovative approach employs hierarchical and integrated features to alleviate recognition degradation and sensitivity to array amplitude and phase errors. Specifically, D $^{2}$ AF-Net contains two fusion-based building blocks: an intra-domain Fusion module mixing local feature and global information, and a staged fusion strategy supporting inter-domain streaming feature interactions and reducing redundant features. Our extensive experimental results based on both simulation and semi-physical datasets show that D $^{2}$ AF-Net achieves state-of-the-art performance on recognition rate, generalization ability in low JNR, and robustness for array errors while requiring fewer or comparable data processing time and interference time than competitive models.
- 2 July 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Aerospace and Electronic Systems
- Vol. PP (00189251) , 1-15
D $^{2}$ AF-Net: A Dual-Domain Adaptive Fusion Method for Radar Deception Jamming RecognitionDeception jamming has garnered widespread attention for threatening radar systems due to its lower energy requirement compared to traditional suppression jamming. With the rapid evolution of electronic warfare technologies, multiple-input multiple-output (MIMO) radar has demonstrated remarkable progress in countering jamming. Non-networked deception jammers, capable of generating compound deception jamming within MIMO radar's main lobe, present significant challenges. However, existing jamming recognition methods suffer from static assumptions tailored to limited jamming patterns, excessive dependence on prior knowledge, inefficacy in extracting discriminative features, and high computational cost. To overcome these limitations, we first propose the dual-domain adaptive fusion network (D $^{2}$ AF-Net). This innovative approach employs hierarchical and integrated features to alleviate recognition degradation and sensitivity to array amplitude and phase errors. Specifically, D $^{2}$ AF-Net contains two fusion-based building blocks: an intra-domain Fusion module mixing local feature and global information, and a staged fusion strategy supporting inter-domain streaming feature interactions and reducing redundant features. Our extensive experimental results based on both simulation and semi-physical datasets show that D $^{2}$ AF-Net achieves state-of-the-art performance on recognition rate, generalization ability in low JNR, and robustness for array errors while requiring fewer or comparable data processing time and interference time than competitive models.
- 20 February 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Signal Processing
- Vol. 73 (1053587X) , 1309-1322
D2CN: Distributed Deep Convolutional Network
With the rapid growth of distributed systems, deep learning-based multi-source data processing has drawn extensive attention, especially for the multi-channel networks. However, the conventional ones lack a strong theoretical foundation and the data in each channel lack necessary interactions, giving rise to insufficient robustness. Here we derive a network termed as distributed deep convolutional network (D2CN) to overcome this issue, which is explained by integrating generalized singular value decomposition (GSVD) with the principles of Hankel convolution framelet. Specifically, we employ the feature extraction capability of GSVD to perform data interactions by forward/backward propagation, where numerous inputs are designed using the common bases and reliable performance is achieved by training a shared set of right bases. We go over the network’s scalability to show its benefits in performance and robustness. Moreover, we show that the encoder-decoder scheme allows the network suitable for a wide range of inverse situations. Finally, we demonstrate the superiority of the D2CN over other fundamental networks through numerical experiments conducted on classical image denoising.
- 20 February 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Signal Processing
- Vol. 73 (1053587X) , 1309-1322
D2CN: Distributed Deep Convolutional Network
With the rapid growth of distributed systems, deep learning-based multi-source data processing has drawn extensive attention, especially for the multi-channel networks. However, the conventional ones lack a strong theoretical foundation and the data in each channel lack necessary interactions, giving rise to insufficient robustness. Here we derive a network termed as distributed deep convolutional network (D2CN) to overcome this issue, which is explained by integrating generalized singular value decomposition (GSVD) with the principles of Hankel convolution framelet. Specifically, we employ the feature extraction capability of GSVD to perform data interactions by forward/backward propagation, where numerous inputs are designed using the common bases and reliable performance is achieved by training a shared set of right bases. We go over the network’s scalability to show its benefits in performance and robustness. Moreover, we show that the encoder-decoder scheme allows the network suitable for a wide range of inverse situations. Finally, we demonstrate the superiority of the D2CN over other fundamental networks through numerical experiments conducted on classical image denoising.
- 25 April 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Geoscience and Remote Sensing
- Vol. 62 (01962892) , 1-14
Unsupervised Pose Decoder: Learn to Disentangle the Pose Attribute for Point Cloud Shape Analysis
Pose is a fundamental attribute of 3-D point cloud shape, which profoundly impacts point cloud analysis tasks. However, it is very tricky to directly solve the pose attribute, since it is deeply coupled with geometry shape. To this end, the representation separation strategy has been proposed, where the global representation is modeled as a combination of the pose-related part representation and the geometry shape part representation. However, these methods still cannot model the representation of the pose attribute well. As a reply, we design a new pose decoder in this article, learning to disentangle the pose attribute by exploiting its complement, i.e., the geometry shape part representation. Specifically, a Siamese structure is introduced constituting of two shared branches, where two consistent point clouds with different pose attributes are input. The geometry shape part representation and the global representation are learned in each branch network to solve the pose-related part representation for disentangling the pose distribution. Then, we emphasize the completeness and no redundancy of geometry shape part representation by designing two constraints: 1) we recover the learned geometry shape part representation to a point cloud and enforce it to maintain the same geometry shape as the original input point cloud to guarantee all geometry shape information is retained and 2) we develop two geometry shape part representations embedded from two branches to be the same so as to filter the pose information out. These two constraints are incorporated into the unsupervised loss function to train our pose decoder. Our pose decoder can be integrated into different point cloud shape analysis methods. We evaluate our pose decoder in point cloud classification and part segmentation tasks to handle the pose diversity problem of the input point cloud, which significantly improves the robustness. Besides, the obtained respective poses of input point clouds can be used to register them naturally, making the unsupervised method achieving superior performance.
- 25 April 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Geoscience and Remote Sensing
- Vol. 62 (01962892) , 1-14
Unsupervised Pose Decoder: Learn to Disentangle the Pose Attribute for Point Cloud Shape Analysis
Pose is a fundamental attribute of 3-D point cloud shape, which profoundly impacts point cloud analysis tasks. However, it is very tricky to directly solve the pose attribute, since it is deeply coupled with geometry shape. To this end, the representation separation strategy has been proposed, where the global representation is modeled as a combination of the pose-related part representation and the geometry shape part representation. However, these methods still cannot model the representation of the pose attribute well. As a reply, we design a new pose decoder in this article, learning to disentangle the pose attribute by exploiting its complement, i.e., the geometry shape part representation. Specifically, a Siamese structure is introduced constituting of two shared branches, where two consistent point clouds with different pose attributes are input. The geometry shape part representation and the global representation are learned in each branch network to solve the pose-related part representation for disentangling the pose distribution. Then, we emphasize the completeness and no redundancy of geometry shape part representation by designing two constraints: 1) we recover the learned geometry shape part representation to a point cloud and enforce it to maintain the same geometry shape as the original input point cloud to guarantee all geometry shape information is retained and 2) we develop two geometry shape part representations embedded from two branches to be the same so as to filter the pose information out. These two constraints are incorporated into the unsupervised loss function to train our pose decoder. Our pose decoder can be integrated into different point cloud shape analysis methods. We evaluate our pose decoder in point cloud classification and part segmentation tasks to handle the pose diversity problem of the input point cloud, which significantly improves the robustness. Besides, the obtained respective poses of input point clouds can be used to register them naturally, making the unsupervised method achieving superior performance.
- 1 August 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 32 (12) , 8462-8475
Learning a Task-Specific Descriptor for Robust Matching of 3D Point Clouds
Existing learning-based point feature descriptors are usually task-agnostic, which pursue describing the individual 3D point clouds as accurate as possible. However, the matching task aims at describing the corresponding points consistently across different 3D point clouds. Therefore these too accurate features may play a counterproductive role due to the inconsistent point feature representations of correspondences caused by the unpredictable noise, partiality, deformation, etc., in the local geometry. In this paper, we propose to learn a robust task-specific feature descriptor to consistently describe the correct point correspondence under interference. Born with an Encoder and a Dynamic Fusion module, our method EDFNet develops from two aspects. First, we augment the matchability of correspondences by utilizing their repetitive local structure. To this end, a special encoder is designed to exploit two input point clouds jointly for each point descriptor. It not only captures the local geometry of each point in the current point cloud by convolution, but also exploits the repetitive structure from paired point cloud by Transformer. Second, we propose a dynamical fusion module to jointly use different scale features. There is an inevitable struggle between robustness and discriminativeness of the single scale feature. Specifically, the small scale feature is robust since little interference exists in this small receptive field. But it is not sufficiently discriminative as there are many repetitive local structures within a point cloud. Thus the resultant descriptors will lead to many incorrect matches. In contrast, the large scale feature is more discriminative by integrating more neighborhood information. But it is easier to be disturbed since there is much more interference in the large receptive field. Compared with the conventional fusion strategy that handles multiple scale features equally, we analyze the consistency of them to judge the clean ones and perform larger aggregation weights on them during fusion. Then, a robust and discriminative feature descriptor is achieved by focusing on multiple clean scale features. Extensive evaluations validate that EDFNet learns a task-specific descriptor, which achieves state-of-the-art or comparable performance for robust matching of 3D point clouds.
- 1 August 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 32 (12) , 8462-8475
Learning a Task-Specific Descriptor for Robust Matching of 3D Point Clouds
Existing learning-based point feature descriptors are usually task-agnostic, which pursue describing the individual 3D point clouds as accurate as possible. However, the matching task aims at describing the corresponding points consistently across different 3D point clouds. Therefore these too accurate features may play a counterproductive role due to the inconsistent point feature representations of correspondences caused by the unpredictable noise, partiality, deformation, etc., in the local geometry. In this paper, we propose to learn a robust task-specific feature descriptor to consistently describe the correct point correspondence under interference. Born with an Encoder and a Dynamic Fusion module, our method EDFNet develops from two aspects. First, we augment the matchability of correspondences by utilizing their repetitive local structure. To this end, a special encoder is designed to exploit two input point clouds jointly for each point descriptor. It not only captures the local geometry of each point in the current point cloud by convolution, but also exploits the repetitive structure from paired point cloud by Transformer. Second, we propose a dynamical fusion module to jointly use different scale features. There is an inevitable struggle between robustness and discriminativeness of the single scale feature. Specifically, the small scale feature is robust since little interference exists in this small receptive field. But it is not sufficiently discriminative as there are many repetitive local structures within a point cloud. Thus the resultant descriptors will lead to many incorrect matches. In contrast, the large scale feature is more discriminative by integrating more neighborhood information. But it is easier to be disturbed since there is much more interference in the large receptive field. Compared with the conventional fusion strategy that handles multiple scale features equally, we analyze the consistency of them to judge the clean ones and perform larger aggregation weights on them during fusion. Then, a robust and discriminative feature descriptor is achieved by focusing on multiple clean scale features. Extensive evaluations validate that EDFNet learns a task-specific descriptor, which achieves state-of-the-art or comparable performance for robust matching of 3D point clouds.
- 5 May 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Signal Processing Letters
- Vol. 29 (10709908) , 1192-1196
Searching Dense Point Correspondences via Permutation Matrix Learning
Although 3D point cloud data has received widespread attentions as a general form of 3D signal expression, applying point clouds to the task of dense correspondence estimation between 3D shapes has not been investigated widely. Furthermore, even in the few existing 3D point cloud-based methods, an important and widely acknowledged principle, i.e . one-to-one matching, is usually ignored. In response, this paper presents a novel end-to-end learning-based method to estimate the dense correspondence of 3D point clouds, in which the problem of point matching is formulated as a zero-one assignment problem to achieve a permutation matching matrix to implement the one-to-one principle fundamentally. Note that the classical solutions of this assignment problem are always non-differentiable, which is fatal for deep learning frameworks. Thus we design a special matching module, which solves a doubly stochastic matrix at first and then projects this obtained approximate solution to the desired permutation matrix. Moreover, to guarantee end-to-end learning and the accuracy of the calculated loss, we calculate the loss from the learned permutation matrix but propagate the gradient to the doubly stochastic matrix directly which bypasses the permutation matrix during the backward propagation. Our method can be applied to both non-rigid and rigid 3D point cloud data and extensive experiments show that our method achieves state-of-the-art performance for dense correspondence learning. The code will be released.
- 5 May 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Signal Processing Letters
- Vol. 29 (10709908) , 1192-1196
Searching Dense Point Correspondences via Permutation Matrix Learning
Although 3D point cloud data has received widespread attentions as a general form of 3D signal expression, applying point clouds to the task of dense correspondence estimation between 3D shapes has not been investigated widely. Furthermore, even in the few existing 3D point cloud-based methods, an important and widely acknowledged principle, i.e . one-to-one matching, is usually ignored. In response, this paper presents a novel end-to-end learning-based method to estimate the dense correspondence of 3D point clouds, in which the problem of point matching is formulated as a zero-one assignment problem to achieve a permutation matching matrix to implement the one-to-one principle fundamentally. Note that the classical solutions of this assignment problem are always non-differentiable, which is fatal for deep learning frameworks. Thus we design a special matching module, which solves a doubly stochastic matrix at first and then projects this obtained approximate solution to the desired permutation matrix. Moreover, to guarantee end-to-end learning and the accuracy of the calculated loss, we calculate the loss from the learned permutation matrix but propagate the gradient to the doubly stochastic matrix directly which bypasses the permutation matrix during the backward propagation. Our method can be applied to both non-rigid and rigid 3D point cloud data and extensive experiments show that our method achieves state-of-the-art performance for dense correspondence learning. The code will be released.
- 14 January 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 32 (8) , 4997-5010
VRNet: Learning the Rectified Virtual Corresponding Points for 3D Point Cloud Registration
3D point cloud registration is fragile to outliers, which are labeled as the points without corresponding points. To handle this problem, a widely adopted strategy is to estimate the relative pose based only on some accurate correspondences, which is achieved by building correspondences on the identified inliers or by selecting reliable ones. However, these approaches are usually complicated and time-consuming. By contrast, the virtual point-based methods learn the virtual corresponding points (VCPs) for all source points uniformly without distinguishing the outliers and the inliers. Although this strategy is time-efficient, the learned VCPs usually exhibit serious collapse degeneration due to insufficient supervision and the inherent distribution limitation. In this paper, we propose to exploit the best of both worlds and present a novel robust 3D point cloud registration framework. We follow the idea of the virtual point-based methods but learn a new type of virtual points called rectified virtual corresponding points (RCPs), which are defined as the point set with the same shape as the source and with the same pose as the target . Hence, a pair of consistent point clouds, i.e . source and RCPs, is formed by rectifying VCPs to RCPs (VRNet), through which reliable correspondences between source and RCPs can be accurately obtained. Since the relative pose between source and RCPs is the same as the relative pose between source and target , the input point clouds can be registered naturally. Specifically, we first construct the initial VCPs by using an estimated soft matching matrix to perform a weighted average on the target points. Then, we design a correction-walk module to learn an offset to rectify VCPs to RCPs, which effectively breaks the distribution limitation of VCPs. Finally, we develop a hybrid loss function to enforce the shape and geometry structure consistency of the learned RCPs and the source to provide sufficient supervision. Extensive experiments on several benchmark datasets demonstrate that our method achieves advanced registration performance and time-efficiency simultaneously. The code will be made public.
- 14 January 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 32 (8) , 4997-5010
VRNet: Learning the Rectified Virtual Corresponding Points for 3D Point Cloud Registration
3D point cloud registration is fragile to outliers, which are labeled as the points without corresponding points. To handle this problem, a widely adopted strategy is to estimate the relative pose based only on some accurate correspondences, which is achieved by building correspondences on the identified inliers or by selecting reliable ones. However, these approaches are usually complicated and time-consuming. By contrast, the virtual point-based methods learn the virtual corresponding points (VCPs) for all source points uniformly without distinguishing the outliers and the inliers. Although this strategy is time-efficient, the learned VCPs usually exhibit serious collapse degeneration due to insufficient supervision and the inherent distribution limitation. In this paper, we propose to exploit the best of both worlds and present a novel robust 3D point cloud registration framework. We follow the idea of the virtual point-based methods but learn a new type of virtual points called rectified virtual corresponding points (RCPs), which are defined as the point set with the same shape as the source and with the same pose as the target . Hence, a pair of consistent point clouds, i.e . source and RCPs, is formed by rectifying VCPs to RCPs (VRNet), through which reliable correspondences between source and RCPs can be accurately obtained. Since the relative pose between source and RCPs is the same as the relative pose between source and target , the input point clouds can be registered naturally. Specifically, we first construct the initial VCPs by using an estimated soft matching matrix to perform a weighted average on the target points. Then, we design a correction-walk module to learn an offset to rectify VCPs to RCPs, which effectively breaks the distribution limitation of VCPs. Finally, we develop a hybrid loss function to enforce the shape and geometry structure consistency of the learned RCPs and the source to provide sufficient supervision. Extensive experiments on several benchmark datasets demonstrate that our method achieves advanced registration performance and time-efficiency simultaneously. The code will be made public.