Zhen Shu

Semantic segmentation of airborne laser scanning (ALS) point clouds is a valuable yet challenging task in remote sensing. When processing large-scale ALS scenes, it is necessary to partition them into smaller blocks for ease of handling. However, this partitioning introduces a challenge in capturing the ample spatial context within each block to adequately recognize the objects with a significant spatial span. This limitation becomes particularly pronounced when relying solely on the 3-D representations as the input of neural networks. To incorporate sufficient contextual information in ALS data semantic segmentation, we propose a multimodal-based segmentation framework called projection-based context embedding (PCE) in this study. PCE effectively combines the advantages of 2-D image and 3-D point-voxel representations, which are the computational efficiency and the representation capability for fine-grained 3-D geometries. The 2-D projection is used to encode a large-scale semantic context, which is computationally expensive to be obtained using only pure 3-D representation. Simultaneously, the sparse-point-voxel convolution (SPVConv) is employed to focus on learning 3-D features from a small block of points centered on the large-scale context. Finally, to fully exploit the power of each modality, the embedding disentangling (ED) strategy is proposed additionally to combine the context embedding from the 2-D image with 3-D features for the final prediction. We demonstrate the state-of-the-art performance of PCE through extensive experiments on public large-scale ALS point cloud datasets.

Automatic extraction of vector polygons of buildings from remotely sensed images is an important but difficult task. Recent existing methods based on deep learning usually adopt a multi-stage solution of semantic segmentation, contour detection, and polygon simplification. Such a long processing chain may lead to unreliable results as the boundary regularization and optimization processes are ultimately completed by utilizing low-level features, which ignores the potential of deep features in polygon generation. In this paper, we present an algorithm for directly extracting simplified polygons of buildings in remotely sensed images. The key of this task is the encoding of the polygon structure. PolyMapper [1] utilizes a recurrent neural network (RNN) to produce vertices of a polygon sequentially. Due to the limitation of RNN, this approach is unstable and difficult to deal with objects with complex shapes. In this work, we encode the polygon into a tensor representation and utilize a non-recurrent manner to recover the polygon structure. In our algorithm, two types of points are utilized, i.e., the corner point and the connecting point. Corner points are utilized to delineate the building outlines and form the vertices of the final polygon. Meanwhile, connecting points are sampled from the edges of the buildings for the assistance of the connection of the corner points. Furthermore, we predict the forward and backward directions of each keypoint in a polygon and propose a bidirectional tracing strategy for the polygon structure recovery. Our approach is simple, effective and robust. Experiments on public datasets demonstrate the superiority of the proposed algorithm. The code is made publicly available at https://github.com/sz94/bldvec.

Ground filtering (GF) is a fundamental step for airborne laser scanning (ALS) data processing. The advent of deep learning techniques provides new solutions to this problem. Existing deep-learning-based methods utilize a segmentation or classification framework to extract ground/non-ground points, which suffers from a dilemma in keeping high spatial resolution while acquiring rich contextual information when dealing with large-scale ALS data due to the computing resource limits. To this end, we propose SeqGP, a novel deep-learning-based GF pipeline that explicitly converts the GF task into an iterative sequential ground prediction (SeqGP) problem using points-profiles. The proposed SeqGP utilizes deep reinforcement learning (DRL) to optimize the prediction sequence and retrieve the bare terrain gradually. The 3D sparse convolution is integrated with the SeqGP strategy to generate high-precision classification results with memory efficiency. Extensive experiments on two challenging test sets demonstrate the state-of-the-art filtering performance and universality of the proposed method in dealing with large-scale ALS data.

Accurate building extraction from remotely sensed images is essential for topographic mapping, cadastral surveying and many other applications. Fully automatic segmentation methods still remain a great challenge due to the poor generalization ability and the inaccurate segmentation results. In this work, we are committed to robust click-based interactive building extraction in remote sensing imagery. We argue that stability is vital to an interactive segmentation system, and we observe that the distance of the newly added click to the boundaries of the previous segmentation mask contains progress guidance information of the interactive segmentation process. To promote the robustness of the interactive segmentation, we exploit this information with the previous segmentation mask, positive and negative clicks to form a progress guidance map, and feed it to a convolutional neural network (CNN) with the original RGB image, we name the network as PGR-Net. In addition, an adaptive zoom-in strategy and an iterative training scheme are proposed to further promote the stability of PGR-Net. Compared with the latest methods FCA and f-BRS, the proposed PGR-Net basically requires 1–2 fewer clicks to achieve the same segmentation results. Comprehensive experiments have demonstrated that the PGR-Net outperforms related state-of-the-art methods on five natural image datasets and three building datasets of remote sensing images.

Automatic extraction of region objects from high-resolution satellite imagery presents a great challenge, because there may be very large variations of the objects in terms of their size, texture, shape, and contextual complexity in the image. To handle these issues, we present a novel, deep-learning-based approach to interactively extract non-artificial region objects, such as water bodies, woodland, farmland, etc., from high-resolution satellite imagery. First, our algorithm transforms user-provided positive and negative clicks or scribbles into guidance maps, which consist of a relevance map modified from Euclidean distance maps, two geodesic distance maps (for positive and negative, respectively), and a sampling map. Then, feature maps are extracted by applying a VGG convolutional neural network pre-trained on the ImageNet dataset to the image X, and they are then upsampled to the resolution of X. Image X, guidance maps, and feature maps are integrated as the input tensor. We feed the proposed attention-guided, multi-scale segmentation neural network (AGMSSeg-Net) with the input tensor above to obtain the mask that assigns a binary label to each pixel. After a post-processing operation based on a fully connected Conditional Random Field (CRF), we extract the selected object boundary from the segmentation result. Experiments were conducted on two typical datasets with diverse region object types from complex scenes. The results demonstrate the effectiveness of the proposed method, and our approach outperforms existing methods for interactive image segmentation.

For automatic building detection in aerial images, small and dense buildings make it a very challenging task. It is because small objects lack sufficient information, and dense building distribution makes the localization of the objects confusing. High-quality building proposals can certainly promote the detection performance. The key to the problem is adopting sufficiently proper size and location of bounding boxes to use the image information for the proposal generation. Based on machine learning with a deep convolutional neural network, this letter proposes a new pipeline of building proposal generation, which is an end-to-end process during training and testing. First, the proposed pipeline attempts to find possible object center points called point proposals. Subsequently, a location refinement module and an object scoring module are applied to the boxes generated from the point proposals with a series of sizes and aspect ratios to obtain the final object proposals. This center-point-guided location refinement and multibox scoring method effectively alleviates the small and dense object problems. Experiments in INRIA Aerial Image Labeling data set demonstrate the better performance of our approach than other state-of-the-art proposal methods. In addition, we add a normal classification branch based on our generated proposals to conduct experiments on detection task. Detection result outperforms the latest detection framework R-FCN equipped with ResNet-101 7% mean average precision at 0.7.