Zhijie Zheng

The 3D object detector usually uses a framework similar to 2D detection and benefits from the advancements of 2D detection tasks. In these frameworks, it is necessary to make the unstructured, sparse point cloud features into dense grids to be compatible with popular 2D operators such as convolution and transformers, which also causes extra computational costs. In this paper, we propose a simple and efficient Fully Sparse TRansformer (FSTR) for LiDAR-based 3D object detection, which is able to combine with state-of-the-art sparse backbones to form a fully sparse, end-to-end, simple, and efficient detection framework. FSTR uses the sparse voxel feature from the sparse backbone as the input token without any custom operators. Further, we introduce the dynamic queries to provide a priori location and context of the foreground for the decoder and drop the high-confidence background tokens to further reduce redundant computations. We propose Gaussian denoising queries to speed up the decoder training and make it more adaptable to the distribution of sparse voxel features. Extensive experiments on the nuScenes benchmark and the Argoverse2 benchmark validate the effectiveness of the proposed method. FSTR outperforms all LiDAR real-time methods by 69.5 mAP and 72.9 NDS on the official benchmark of nuScenes dataset. On the long-range detection benchmark Argoverse2, the proposed method achieves a new state-of-art performance of 39.9 mAP which outperforms the existing LiDAR detectors, even the LiDAR-Camera detectors by a large margin (+9.4 mAP and +7.5mAP), showing the great advantage of the proposed method for long-range detection.

For fine-grained human perception tasks such as pose estimation and activity recognition, radar-based sensors show advantages over optical cameras in low-visibility, privacy-aware, and wall-occlusive environments. Radar transmits radio frequency signals to irradiate the target of interest and store the target information in the echo signals. One common approach is to transform the echoes into radar images and extract the features with convolutional neural networks. This article introduces RadarFormer, the first method that introduces the self-attention (SA) mechanism to perform human perception tasks directly from radar echoes. It bypasses the imaging algorithm and realizes end-to-end signal processing. Specifically, we give constructive proof that processing radar echoes using the SA mechanism is at least as expressive as processing radar images using the convolutional layer. On this foundation, we design RadarFormer, which is a Transformer-like model to process radar signals. It benefits from the fast-/slow-time SA mechanism considering the physical characteristics of radar signals. RadarFormer extracts human representations from radar echoes and handles various downstream human perception tasks. The experimental results demonstrate that our method outperforms the state-of-the-art radar-based methods both in performance and computational cost and obtains accurate human perception results even in dark and occlusive environments.

This article proposes a 3-D human pose estimation method using through-wall radar (TWR) systems, which extends and supplements new applications in the era of the Internet of Things (IoT). TWR system can penetrate nonmetallic obstacles and perceive wall-occlusive human targets, but the physical characteristics of radio frequency (RF) signals, such as poor imaging resolution and specularity effect, make the pose estimation process highly ill-posed. In this work, we propose a mutual information maximizing deeply supervised network (MIMDSN), which aims to extract accurate and robust 3-D human skeletons from TWR images. Inspired by past works, an optical system is attached to the TWR system to provide cross-modal pseudo labels. Based on a depth design philosophy of convolutional neural networks that meets radar resolution constraints, we design a resolution-guided pose estimation network for keypoint coordinate regression. To alleviate the ill-posed problem, supervising solely the network output is insufficient. The cross-modal supervision is not only built on predictions, but also on features of the network’s hidden layer. With the help of information theory, the mutual information between features and pseudo labels is maximized for feature alignment and discriminability enhancement. Experiments show competitive performance against state-of-the-art RF-based human pose estimation methods and can reconstruct accurate 3-D skeletons in multitarget, low-visibility, and wall-occlusive scenes.

Ultra-wideband (UWB) multiple-input multiple-output (MIMO) through-wall radar is widely used in through-wall human target detection for its good penetration characteristics and resolution. However, in actual detection scenarios, weak target masking and adjacent target unresolving will occur in through-wall imaging due to factors such as resolution limitations and differences in human reflectance, which will reduce the probability of target detection. An improved U-Net model is proposed in this paper to improve the detection probability of through-wall targets. In the proposed detection method, a ResNet module and a squeeze-and-excitation (SE) module are integrated in the traditional U-Net model. The ResNet module can reduce the difficulty of feature learning and improve the accuracy of detection. The SE module allows the network to perform feature recalibration and learn to use global information to emphasize useful features selectively and suppress less useful features. The effectiveness of the proposed method is verified via simulations and experiments. Compared with the order statistics constant false alarm rate (OS-CFAR), the fully convolutional networks (FCN) and the traditional U-Net, the proposed method can detect through-wall weak targets and adjacent unresolving targets effectively. The detection precision of the through-wall target is improved, and the missed detection rate is minimized.

Through-wall radar (TWR) can image the target of interest and capture the human sensing information. However, the poor human interpretability of TWR images and the lack of effective supervision make the extraction of complete body contour intractable. This letter proposes dual UNet, an unsupervised human contour extraction method for TWR images. Specifically, the method adopts two UNets with the same structure. One serves as the encoder to convert the TWR images into the latent representation. Another serves as the decoder to reconstruct the latent representation into the original images. Reconstruction loss and smooth normalized cut loss are optimized together to offset the dependence on labels and supplement global segment constraints. After training and post-processing, the latent representation can be used as the result of contour extraction. Experimental results show that dual UNet stands out among unsupervised human contour extraction methods in both free space and wall-occlusive scenarios, opening the possibility of learning useful human sensing information from raw TWR images without manual annotations.

Existing unsupervised domain adaptive (UDA) 3D detection methods only address the domain gap caused by the prior size of 3D bounding boxes between different datasets, which ignore the difference in the distribution of point clouds. To address this challenge, we propose an unsupervised domain adaptive 3D detection by data adaption, which trains the model by transferring the source domain instances into the target domain scenes by adaptive point distribution. First, an instance transferring method is proposed for selecting and transferring suitable instances from the source domain into the target domain scene; Second, we propose an adaptive downsampling method to adjust the point cloud distribution of the transferred instances to approximate the points distribution of the target domain. Finally, our method trains the randomly initialized detector with the pseudo-instances in the target domain. To the best of our knowledge, we first address the UDA problem of the 3D detectors from the perspective of data. Extensive experiments on several popular datasets show that the proposed method outperforms the existing state-of-the-art methods by a large margin. Further experiments also show our approach is detector-agnostic and achieves consistent and significant gains on all types of 3D detectors.

Recent through-wall radar (TWR) systems can reconstruct the pose of human targets blocked by occlusion. They rely on the fusion of optical and radar data to avoid the painful annotation burden. However, the fusion process is not always reliable, especially for human joint coordinates that carry 3-D spatial information. Inspired by cross-modal learning and self-supervised learning, this letter proposes a two-stage 3-D human pose reconstruction method for TWR systems. In the cross-modal supervision stage, the pretrained optical model provides initial noisy labels extracted from optical images. In the self-supervision stage, supervised labels and the model weight are corrected circularly with radar images. The self-supervision enhances the robustness of the model and the reliability of labels. It can be directly extended to existing radar-based pose reconstruction methods, and hardly requires extra training time. Experiments show the model beats state of the art (SOTA) for reconstructing 3-D poses from TWR images and contains robust generalization in unseen wall-occlusive scenes.

Ultra-wideband (UWB) impulse radar is widely used for through-wall human detection due to its high-range resolution and high penetration capability. UWB impulse radar can detect human targets in non-line-of-sight (NLOS) conditions, mainly based on the chest motion caused by human respiration. The automatic detection and extraction of multiple stationary human targets is still a challenge. Missed alarms often exist if the detection method is based on the energy of the human target. This is mainly because factors such as the range of the target, the intensity of the respiratory movement, and the shadow effect will make a difference between the energy scattered by targets. Weak targets are easily masked by strong targets and thus cannot be detected. Therefore, in this paper, a multiple stationary human targets detection method based on convolutional neural network (CNN) in through-wall UWB impulse radar is proposed. After performing the signal-to-clutter-and-noise ratio (SCNR) enhancement method on the raw radar data, the range-slow-time matrix is fed into a six-layer CNN. Benefiting from the powerful feature extraction capability of CNN, the target point of interest (TPOI) can be extracted from the data matrix. The clustering algorithm is used to simplify the TPOIs to achieve accurate detection of multiple targets behind the wall. The effectiveness of the proposed method is verified by the experimental data.

Although the through-the-wall radar imaging (TWRI) system working in the appropriate frequency band can penetrate the nonmetallic obstacles and sense the targets behind, its low imaging spatial resolution hinders the acquisition of more detailed information, such as human pose and shape. This article mainly discusses a deep learning-based human pose and shape recovery method from TWRI images. Inspired by cross-modal learning, the method follows a teacher–student learning pipeline that avoids the heavy cost of manual labeling. Specifically, a camera is attached to the self-develop radar system to simultaneously capture paired red-green-blue (RGB) images and TWRI images in a scenario without wall occlusion. A pose estimation framework (Hourglass) and a semantic segmentation framework (UNet) serve as the teacher network to convert the RGB images into the pose keypoints and the shape masks. By taking inspiration from the topological architecture of these frameworks, a student network radar pose shape network (RPSNet) is designed to extract the information from the corresponding radar images and predict the keypoints and masks that are close to the results above. Instead of learning two single-task objectives independently, multitasking learning is introduced to adaptatively learn common features. When applied to wall-occlusive scenarios, only the radar images are collected and fed into the student network for pose and shape recovery. The advantages of this method over computer vision-based methods for human recovery are demonstrated in scenarios both without and with wall occlusion.

Clutter removal in ground-penetrating radar (GPR) B-scan data has been widely studied in recent years. In this letter, we propose a novel data-driven clutter suppression method in GPR data based on conditional generative adversarial nets (cGANs). The proposed method learns a function that maps the cluttered data to the clutter-free data from the training set. The training set consists of pairs of cluttered data and corresponding clutter-free data. Different from the traditional method that only uses the simulation training set, we simulate the clutter-free data and add the real collected non-target data to the simulated clutter-free data as cluttered data, so that the trained network can generalize well to the real GPR data. The proposed method is compared with the subspace method, sparse representation-based method, and low-rank and sparse matrix decomposition (LRSD) methods on both simulation data and real collected data. The results show that the proposed method has higher performance in terms of computational complexity, clutter suppression results, and applicability than those state-of-the-art methods.