INVITATION: A Framework for Enhancing UAV Image Semantic Segmentation Accuracy Through Depth Information Fusion
- 30 January 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Geoscience and Remote Sensing Letters
- Vol. 22 (1545598X) , 1-5
Abstract
With the increasing use of Unmanned Aerial Vehicles (UAVs), improving the accuracy of semantic segmentation is becoming critical. Depth information preserves geometric structure, serving as an invaluable supplement to color-rich UAV imagery. Inspired by this, we proposed a novel framework named INVITATION, which exclusively takes original UAV imagery as input, yet is capable of obtaining complemented depth information and fusing into RGB semantic segmentation models effectively, thereby enhancing UAV semantic segmentation accuracy. Concretely, this framework supports two distinct depth generation approaches: high-precision Multi-View Stereo (MVS) depth reconstruction using multiple views or video sequences via Structure from Motion (SfM), and monocular depth estimation using individual images. Our empirical evaluations conducted on the UAVid dataset showed that mIoU metric of INVITATION utilized precise reconstructed depth maps via MVS improved from 66.02% to 70.57%, while utilized depth predictions from pre-trained models reached 69.69%, which supports the effectiveness of extracting and fusing depth information from original imagery in enhancing UAV semantic segmentation. This study explores a novel approach to acquire UAV multi-modal information at low data cost, highlights the advantages of incorporating depth information into UAV semantic analysis, and paves the way for further studies on the integration of multi-modal UAV information. Our code is available at https://github.com/CVEO/INVITATION.Keywords
Funding Information
- China Postdoctoral Science Foundation (No. 2020M680109)
- National Natural Science Foundation of China (No. 42101346)
- Science and Technology Commission of Shanghai Municipality (No. 22DZ1100800)
This publication has 13 references indexed in Scilit:
- Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data2024
- UNetFormer: A UNet-like transformer for efficient semantic segmentation of remote sensing urban scene imageryISPRS Journal of Photogrammetry and Remote Sensing, 2022
- Light-Weight Semantic Segmentation Network for UAV Remote Sensing ImagesIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021
- Object-Based Classification Framework of Remote Sensing Images With Graph Convolutional NetworksIEEE Geoscience and Remote Sensing Letters, 2021
- UAV assistance paradigm: State-of-the-art in applications and challengesJournal of Network and Computer Applications, 2020
- UAVid: A semantic segmentation dataset for UAV imageryISPRS Journal of Photogrammetry and Remote Sensing, 2020
- Digging Into Self-Supervised Monocular Depth Estimation2019
- Symmetrical Dense-Shortcut Deep Fully Convolutional Networks for Semantic Segmentation of Very-High-Resolution Remote Sensing ImagesIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2018
- Pyramid Scene Parsing Network2017
- Fully convolutional networks for semantic segmentation2015