Jihao Li
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Jihao Li has a total of 41 co-authors. Here are the top co-authors by number of shared publications:
- Xian Sun
Chinese Academy of Sciences
12 shared publications - Wenkai Zhang
Chinese Academy of Sciences, Aerospace Information Research Institute, Beijing, China
6 shared publications Xin Gao
Aerospace Information Research Institute, Key Laboratory of Target Cognition and Application Technology (TCAT), Chinese Academy of Sciences, Beijing, China
5 shared publications- Yingchao Feng
Chinese Academy of Sciences
5 shared publications - Wenhui Diao
Chinese Academy of Sciences
4 shared publications - Chongyang Li
University of the Chinese Academy of Sciences, Beijing, China
3 shared publications
Latest Publications
- Open Access
- 8 January 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
- Vol. 18  (19391404) , 3905-3920
LMF-Net: A Learnable Multimodal Fusion Network for Semantic Segmentation of Remote Sensing Data
Semantic segmentation of remote sensing images has produced a significant effect on many applications, such as land cover and land use, smoke detection, etc.. With the ever-growing remote sensing data, fusing multi-modal data from different sensors is a feasible and effective scheme for semantic segmentation task. Deep learning technology has prominently promoted the development of semantic segmentation. However, the majority of current approaches commonly focus more on feature mixing and construct relatively complex architectures. The further mining for cross-modal features is comparatively insufficient in heterogeneous data fusion. Additionally, complex structures also lead to relatively heavy computation burden. Therefore, in this paper, we propose an end-to-end Learnable Multi-modal Fusion Network (LMF-Net) for remote sensing semantic segmentation. Concretely, we first develop a Multi-Scale Pooling Fusion (MSPF) module by leveraging pooling operator. It provides key-value pairs with multi-modal complementary information in a parameter-free manner and assigns them to selfattention layers of different modal branches. Then, to further harness the cross-modal collaborative embeddings/features, we elaborate two learnable fusion modules, Learnable Embedding Fusion (LEF) and Learnable Feature Fusion (LFF). They are able to dynamically adjust the collaborative relationships of different modal embeddings and features in a learnable approach, respectively. Experiments on two well-established benchmark datasets reveal that our LMF-Net possesses superior segmentation behavior and strong generalization capability. In terms of computation complexity, it achieves competitive performance as well. Ultimately, the contribution of each component involved in LMF-Net is evaluated and discussed in detail.
- 8 January 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
- Vol. 18  (19391404) , 3905-3920
Open AccessLMF-Net: A Learnable Multimodal Fusion Network for Semantic Segmentation of Remote Sensing Data
Semantic segmentation of remote sensing images has produced a significant effect on many applications, such as land cover and land use, smoke detection, etc.. With the ever-growing remote sensing data, fusing multi-modal data from different sensors is a feasible and effective scheme for semantic segmentation task. Deep learning technology has prominently promoted the development of semantic segmentation. However, the majority of current approaches commonly focus more on feature mixing and construct relatively complex architectures. The further mining for cross-modal features is comparatively insufficient in heterogeneous data fusion. Additionally, complex structures also lead to relatively heavy computation burden. Therefore, in this paper, we propose an end-to-end Learnable Multi-modal Fusion Network (LMF-Net) for remote sensing semantic segmentation. Concretely, we first develop a Multi-Scale Pooling Fusion (MSPF) module by leveraging pooling operator. It provides key-value pairs with multi-modal complementary information in a parameter-free manner and assigns them to selfattention layers of different modal branches. Then, to further harness the cross-modal collaborative embeddings/features, we elaborate two learnable fusion modules, Learnable Embedding Fusion (LEF) and Learnable Feature Fusion (LFF). They are able to dynamically adjust the collaborative relationships of different modal embeddings and features in a learnable approach, respectively. Experiments on two well-established benchmark datasets reveal that our LMF-Net possesses superior segmentation behavior and strong generalization capability. In terms of computation complexity, it achieves competitive performance as well. Ultimately, the contribution of each component involved in LMF-Net is evaluated and discussed in detail.
- 26 August 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Geoscience and Remote Sensing
- Vol. 62  (01962892) , 1-14
Injecting Linguistic Into Visual Backbone: Query-Aware Multimodal Fusion Network for Remote Sensing Visual Grounding
The Remote Sensing Visual Grounding (RSVG) task focuses on accurately identifying and localizing specific targets in remote sensing (RS) images using descriptive query expressions. Existing methods independently extract visual and textual features, ignoring early complementary information between image and text. This leads to information loss and misalignment, limiting the model’s ability to distinguish similar targets. To address this challenge, we propose the Query-Aware Multimodal Fusion Network (QAMFN), which introduces an innovative Query-Guided Visual Attention (QGVA) mechanism in the early stages of the visual encoder. This mechanism integrates textual information during the early visual feature extraction process, thereby resolving the issue of missing image-text complementary information. QGVA ensures that the visual backbone accurately focuses on local features highly relevant to the query by injecting textual information into the visual encoding process. Additionally, to enhance the model’s ability to integrate multimodal information and adapt to more complex RS images, we introduce the Text-Semantic Attention-Guided Masking (TAM) module. TAM aggregates multimodal features processed by the backbones and filters out redundant information, producing high-quality fused features. Experiments demonstrate that our approach sets a new record on the DIOR-RSVG dataset, improving accuracy to 81.67% (an absolute increase of 4.98%).
- 26 August 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Geoscience and Remote Sensing
- Vol. 62  (01962892) , 1-14
Injecting Linguistic Into Visual Backbone: Query-Aware Multimodal Fusion Network for Remote Sensing Visual Grounding
The Remote Sensing Visual Grounding (RSVG) task focuses on accurately identifying and localizing specific targets in remote sensing (RS) images using descriptive query expressions. Existing methods independently extract visual and textual features, ignoring early complementary information between image and text. This leads to information loss and misalignment, limiting the model’s ability to distinguish similar targets. To address this challenge, we propose the Query-Aware Multimodal Fusion Network (QAMFN), which introduces an innovative Query-Guided Visual Attention (QGVA) mechanism in the early stages of the visual encoder. This mechanism integrates textual information during the early visual feature extraction process, thereby resolving the issue of missing image-text complementary information. QGVA ensures that the visual backbone accurately focuses on local features highly relevant to the query by injecting textual information into the visual encoding process. Additionally, to enhance the model’s ability to integrate multimodal information and adapt to more complex RS images, we introduce the Text-Semantic Attention-Guided Masking (TAM) module. TAM aggregates multimodal features processed by the backbones and filters out redundant information, producing high-quality fused features. Experiments demonstrate that our approach sets a new record on the DIOR-RSVG dataset, improving accuracy to 81.67% (an absolute increase of 4.98%).
- Top Cited PapersOpen Access
- 27 February 2024
- Published by Elsevier in Remote Sensing of Environment
- Vol. 305, 114057
GABLE: A first fine-grained 3D building model of China on a national scale from very high resolution satellite imagery
- 27 February 2024
- Published by Elsevier in Remote Sensing of Environment
- Vol. 305, 114057
Top Cited PapersOpen AccessGABLE: A first fine-grained 3D building model of China on a national scale from very high resolution satellite imagery
- 22 September 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Geoscience and Remote Sensing
- Vol. 61  (01962892) , 1-15
Hypersphere-Based Remote Sensing Cross-Modal Text–Image Retrieval via Curriculum Learning
Remote sensing cross-modal text-image retrieval (RSCTIR) is a flexible and human-centered approach to retrieving rich information from different modalities, which has attracted plenty of attention in recent years. It remains challenging because the current methods usually ignore the varying difficulty levels of different sample pairs, stemming from the large image distribution difference and the high text similarity in the remote sensing (RS) field. Therefore, in this paper, we propose an innovative hypersphere-based visual semantic alignment (HVSA) network via curriculum learning. Specifically, we first design an adaptive alignment strategy based on curriculum learning, that aligns RS image-text pairs from easy to hard. Sample pairs with different levels of difficulty are treated unequally, and we obtain a better embedding representation when projecting the features onto the unit hypersphere. Then, to measure the robustness of cross-modal feature alignment on the unit hypersphere, we introduce the feature uniformity strategy. It reduces the occurrence of mismatching cases and improves generalization performance. Finally, we design the key-entity attention (KEA) mechanism to alleviate the problem of information imbalance among different modalities. KEA has the ability to extract information about the key entity which is aligned with textual information. Despite its conciseness, our framework achieves state-of-the-art performance on classical datasets of RSCTIR tasks while enjoying faster inference. The summed recall of HVSA on the RISCD and RSITMD is 120.97 and 198.94, 2.50 and 10.49 points ahead of the current best methods, respectively. Extensive experiments demonstrate the competitiveness of our method. The code has been released at https://github.com/ZhangWeihang99/HVSA.
- 22 September 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Geoscience and Remote Sensing
- Vol. 61  (01962892) , 1-15
Hypersphere-Based Remote Sensing Cross-Modal Text–Image Retrieval via Curriculum Learning
Remote sensing cross-modal text-image retrieval (RSCTIR) is a flexible and human-centered approach to retrieving rich information from different modalities, which has attracted plenty of attention in recent years. It remains challenging because the current methods usually ignore the varying difficulty levels of different sample pairs, stemming from the large image distribution difference and the high text similarity in the remote sensing (RS) field. Therefore, in this paper, we propose an innovative hypersphere-based visual semantic alignment (HVSA) network via curriculum learning. Specifically, we first design an adaptive alignment strategy based on curriculum learning, that aligns RS image-text pairs from easy to hard. Sample pairs with different levels of difficulty are treated unequally, and we obtain a better embedding representation when projecting the features onto the unit hypersphere. Then, to measure the robustness of cross-modal feature alignment on the unit hypersphere, we introduce the feature uniformity strategy. It reduces the occurrence of mismatching cases and improves generalization performance. Finally, we design the key-entity attention (KEA) mechanism to alleviate the problem of information imbalance among different modalities. KEA has the ability to extract information about the key entity which is aligned with textual information. Despite its conciseness, our framework achieves state-of-the-art performance on classical datasets of RSCTIR tasks while enjoying faster inference. The summed recall of HVSA on the RISCD and RSITMD is 120.97 and 198.94, 2.50 and 10.49 points ahead of the current best methods, respectively. Extensive experiments demonstrate the competitiveness of our method. The code has been released at https://github.com/ZhangWeihang99/HVSA.
- Open Access
- 16 August 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
- Vol. 16  (19391404) , 7704-7717
From Plane to Hierarchy: Deformable Transformer for Remote Sensing Image Captioning
With the growth of remote sensing images, un-derstanding image content automatically has attracted many researchers' interests in deep learning for remote sensing image. Inspired from the natural image captioning, the model with CNN-RNN as the backbone and supplemented by attention has been widely used in remote sensing image captioning. However, it is inefficient for the current attention layer to simultaneously mine hidden foreground from the background of remote sensing image and perform feature interactive learning. Meanwhile, the new mainstream language model has recently surpassed the traditional LSTM in sentence generation. For solving the above problems, in this paper, we proposed a novel thought to make the flat remote sensing images stereoscopic by separating the fore- and background. Based on hierarchical image informa-tion, we designed a novel Deformable Transformer equipped with deformable scaled dot-product attention to learn multi-scale feature from fore- and background through the powerful interactive learning ability. Evaluations are conducted on Four classic remote sensing image captioning datasets. Compared with the state-of-the-art methods, our Transformer variant achieves higher captioning accuracy.
- 16 August 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
- Vol. 16  (19391404) , 7704-7717
Open AccessFrom Plane to Hierarchy: Deformable Transformer for Remote Sensing Image Captioning
With the growth of remote sensing images, un-derstanding image content automatically has attracted many researchers' interests in deep learning for remote sensing image. Inspired from the natural image captioning, the model with CNN-RNN as the backbone and supplemented by attention has been widely used in remote sensing image captioning. However, it is inefficient for the current attention layer to simultaneously mine hidden foreground from the background of remote sensing image and perform feature interactive learning. Meanwhile, the new mainstream language model has recently surpassed the traditional LSTM in sentence generation. For solving the above problems, in this paper, we proposed a novel thought to make the flat remote sensing images stereoscopic by separating the fore- and background. Based on hierarchical image informa-tion, we designed a novel Deformable Transformer equipped with deformable scaled dot-product attention to learn multi-scale feature from fore- and background through the powerful interactive learning ability. Evaluations are conducted on Four classic remote sensing image captioning datasets. Compared with the state-of-the-art methods, our Transformer variant achieves higher captioning accuracy.
- 13 April 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Transportation Systems
- Vol. 24  (9) , 9684-9697
GAL: Graph-Induced Adaptive Learning for Weakly Supervised 3D Object Detection
Weakly Supervised 3D Object Detection (WS3DOD) aims to perform 3D object detection with little reliance on 3D labels, which greatly reduces the cost of 3D annotations. In recent literature, the pseudo-label-based approach brings impressive performance, which generates 3D pseudo-labels from 2D bounding boxes. Despite their success, two key issues remain unresolved that reduce the quality of 3D pseudo-labels: 1) the existing local object locating algorithm can not capture complete clusters of points globally, and 2) the existing algorithm can not capture sparse points caused by the unevenly distributed points obtained by LiDAR cameras. Hence, we propose GAL, a Graph-induced Adaptive Learning algorithm, to generate 3D pseudo-labels. First, we propose the Cluster Locating algorithm based on the Minimum Spanning Tree (MST) to globally locate the objects, which can leverage the characteristic that points inside an object are compact while points between objects are discrete. Second, we propose a density-guided adaptive learning algorithm to optimise the Cluster Locating algorithm, named Cuboid Drift. Cuboid Drift considers the inhomogeneous distribution of reflected points on different reflective surfaces of LiDAR imaging. Finally, 3D pseudo-labels generated by GAL are leveraged to train 3D detectors. Extensive experiments on the challenging KITTI and DAIR-V2X-V dataset demonstrate that GAL without 3D labels can be comparable with strongly supervised approaches and outperforms the previous state-of-the-art WS3DOD methods. Moreover, our method saves 88% of the time spent on pseudo-label generation.
- 13 April 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Transportation Systems
- Vol. 24  (9) , 9684-9697
GAL: Graph-Induced Adaptive Learning for Weakly Supervised 3D Object Detection
Weakly Supervised 3D Object Detection (WS3DOD) aims to perform 3D object detection with little reliance on 3D labels, which greatly reduces the cost of 3D annotations. In recent literature, the pseudo-label-based approach brings impressive performance, which generates 3D pseudo-labels from 2D bounding boxes. Despite their success, two key issues remain unresolved that reduce the quality of 3D pseudo-labels: 1) the existing local object locating algorithm can not capture complete clusters of points globally, and 2) the existing algorithm can not capture sparse points caused by the unevenly distributed points obtained by LiDAR cameras. Hence, we propose GAL, a Graph-induced Adaptive Learning algorithm, to generate 3D pseudo-labels. First, we propose the Cluster Locating algorithm based on the Minimum Spanning Tree (MST) to globally locate the objects, which can leverage the characteristic that points inside an object are compact while points between objects are discrete. Second, we propose a density-guided adaptive learning algorithm to optimise the Cluster Locating algorithm, named Cuboid Drift. Cuboid Drift considers the inhomogeneous distribution of reflected points on different reflective surfaces of LiDAR imaging. Finally, 3D pseudo-labels generated by GAL are leveraged to train 3D detectors. Extensive experiments on the challenging KITTI and DAIR-V2X-V dataset demonstrate that GAL without 3D labels can be comparable with strongly supervised approaches and outperforms the previous state-of-the-art WS3DOD methods. Moreover, our method saves 88% of the time spent on pseudo-label generation.
- 7 December 2022
- Published by Elsevier in ISPRS Journal of Photogrammetry and Remote Sensing
- Vol. 195, 233-249
Height aware understanding of remote sensing images based on cross-task interaction
- 7 December 2022
- Published by Elsevier in ISPRS Journal of Photogrammetry and Remote Sensing
- Vol. 195, 233-249
Height aware understanding of remote sensing images based on cross-task interaction
- Open Access
- 9 December 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
- Vol. 15  (19391404) , 339-352
CSF-Net: Color Spectrum Fusion Network for Semantic Labeling of Airborne Laser Scanning Point Cloud
Airborne laser scanning point cloud semantic labeling, which aims to identify the category of each point, plays a significant role in many applications, such as forest observing, powerline extraction, etc. Under the guidance of deep learning technology, the interpretation thought of point clouds has also greatly changed. However, owing to the irregular and unordered natures of point clouds, it is relatively difficult for classification model to distinguish some objects with similar geometry by single-modal data only. Fortunately, additional gain information, e.g., color spectrum which can be complementary to geometric information, is able to effectively promote the classification effect. Therefore, the design of fusion strategy is a critical part in model construction. In this article, aiming to capture more abstract semantic information for color spectrum data, we elaborate a color spectrum fusion (CSF) module. It can be flexibly integrated into a classification pipeline with just negligible parameters. Then, we expand data fusion thoughts for point clouds and color spectrum and investigate three possible fusion strategies. Accordingly, we develop three architectures to construct CSF-Nets. Ultimately, by taking a weighted cross entropy loss, we can train our CSF-Nets in an end-to-end manner. Experiments on two extensively used datasets: Vaihingen 3D and LASDU show that the presented three fusion approaches all can improve the performance, while the earlier fusion strategy performs the best. Besides, compared with other well-performed methods, CSF-Net is still able to achieve satisfactory performance on overall accuracy and m $F_{1}$ -score indicator. This also validates the effectiveness of our multimodal fusion network.
- 9 December 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
- Vol. 15  (19391404) , 339-352
Open AccessCSF-Net: Color Spectrum Fusion Network for Semantic Labeling of Airborne Laser Scanning Point Cloud
Airborne laser scanning point cloud semantic labeling, which aims to identify the category of each point, plays a significant role in many applications, such as forest observing, powerline extraction, etc. Under the guidance of deep learning technology, the interpretation thought of point clouds has also greatly changed. However, owing to the irregular and unordered natures of point clouds, it is relatively difficult for classification model to distinguish some objects with similar geometry by single-modal data only. Fortunately, additional gain information, e.g., color spectrum which can be complementary to geometric information, is able to effectively promote the classification effect. Therefore, the design of fusion strategy is a critical part in model construction. In this article, aiming to capture more abstract semantic information for color spectrum data, we elaborate a color spectrum fusion (CSF) module. It can be flexibly integrated into a classification pipeline with just negligible parameters. Then, we expand data fusion thoughts for point clouds and color spectrum and investigate three possible fusion strategies. Accordingly, we develop three architectures to construct CSF-Nets. Ultimately, by taking a weighted cross entropy loss, we can train our CSF-Nets in an end-to-end manner. Experiments on two extensively used datasets: Vaihingen 3D and LASDU show that the presented three fusion approaches all can improve the performance, while the earlier fusion strategy performs the best. Besides, compared with other well-performed methods, CSF-Net is still able to achieve satisfactory performance on overall accuracy and m $F_{1}$ -score indicator. This also validates the effectiveness of our multimodal fusion network.
- 13 August 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Geoscience and Remote Sensing
- Vol. 60  (01962892) , 1-17
Random Topology and Random Multiscale Mapping: An Automated Design of Multiscale and Lightweight Neural Network for Remote-Sensing Image Recognition
With the proposal of neural architecture search (NAS), automated network architecture design gradually becomes a new way in deep learning research. Due to its high capability regarding automated design, some pioneers have made an attempt to apply NAS in remote sensing and made some achievements, like 1-D/3-D Auto-convolutional neural network (CNN) and polarimetric synthetic aperture radar (PolSAR)-tailored Differentiable Architecture Search (PDAS). However, there are still some areas to be improved for existing NAS in remote-sensing field. In this article, we propose a random topology and random multiscale mapping (RTRMM) method to generate a multiscale and lightweight architecture for remote-sensing image recognition. First, a random topology generator generates the topology through random graph. Second, during the experiment, we find remote-sensing image features extracted by a multiscale network are more appropriate, compared with features extracted by a single-scale model. Nevertheless, the complexity inevitably increases with the introduction of a multiscale concept. Consequently, we design a variable search space consisting of decomposition convolution units under the guidance of mathematical analysis. The mapping of each neuron is then determined by a random multiscale mapping sampler. After that, we assemble the topology and mappings into blocks and construct three RTRMM models. Experiments on four scene classification datasets confirm the feature extraction capability and lightweight performance of RTRMM models. Moreover, we also observe that our approach achieves a better tradeoff between floating-point operations (FLOPs) and accuracy than some current well-behaved methods. Furthermore, the results on Vaihingen dataset verify the high feature-transfer capability.
- 13 August 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Geoscience and Remote Sensing
- Vol. 60  (01962892) , 1-17
Random Topology and Random Multiscale Mapping: An Automated Design of Multiscale and Lightweight Neural Network for Remote-Sensing Image Recognition
With the proposal of neural architecture search (NAS), automated network architecture design gradually becomes a new way in deep learning research. Due to its high capability regarding automated design, some pioneers have made an attempt to apply NAS in remote sensing and made some achievements, like 1-D/3-D Auto-convolutional neural network (CNN) and polarimetric synthetic aperture radar (PolSAR)-tailored Differentiable Architecture Search (PDAS). However, there are still some areas to be improved for existing NAS in remote-sensing field. In this article, we propose a random topology and random multiscale mapping (RTRMM) method to generate a multiscale and lightweight architecture for remote-sensing image recognition. First, a random topology generator generates the topology through random graph. Second, during the experiment, we find remote-sensing image features extracted by a multiscale network are more appropriate, compared with features extracted by a single-scale model. Nevertheless, the complexity inevitably increases with the introduction of a multiscale concept. Consequently, we design a variable search space consisting of decomposition convolution units under the guidance of mathematical analysis. The mapping of each neuron is then determined by a random multiscale mapping sampler. After that, we assemble the topology and mappings into blocks and construct three RTRMM models. Experiments on four scene classification datasets confirm the feature extraction capability and lightweight performance of RTRMM models. Moreover, we also observe that our approach achieves a better tradeoff between floating-point operations (FLOPs) and accuracy than some current well-behaved methods. Furthermore, the results on Vaihingen dataset verify the high feature-transfer capability.