Exploring the Potential of Reconstructed Multispectral Images for Urban Tree Segmentation in Street View Images
Open Access
- 26 June 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
- Vol. 17 (19391404) , 12112-12122
Abstract
Deep learning has gained popularity in recent years for reconstructing hyperspectral and multispectral images, offering cost-effective solutions and promising results. Research on hyperspectral image reconstruction feeds deep learning models with images at specific wavelengths and outputs images in other spectral bands. Although encouraging results of previous works, it should be determined to what extent the reconstructed information can lead to an advantage over the captured images. In this context, the present work inspects whether or not reconstructed spectral images add relevant information to segmentation networks for improving urban tree identification. Specifically, we generate red-edge (ReD) and near-infrared (NIR) images from RGB images using a conditional Generative Adversarial Network (cGAN). The training and validation are carried out with 5770 multispectral images obtained after a custom data augmentation process using an urban hyperspectral dataset. The testing outcomes reveal that ReD and NIR can be generated with an average Structural Similarity Index Measure of 0.93 and 0.88, respectively. Next, the cGAN generates ReD and NIR information of two RGB-based urban tree datasets (i.e., Jekyll, 3949 samples, and Arbocensus, 317 samples). Subsequently, DeepLabV3 and SegFormer segmentation networks are trained, validated, and tested using RGB, RGB+ReD, and RGB+NIR images from Jekyll and Arbocensus datasets. The experiments show that reconstructed multispectral images might not add information to segmentation networks that enhance their performance. Specifically, the p-values from a T-test show no significant difference between the performance of segmentation networks.Keywords
All Related Versions
Funding Information
- Agencia Nacional de Investigación y Desarrollo (11220510, ID21I10360, 21-JD-01)
- Pontificia Universidad Católica de Chile (PIA-2023-VRA-PUC)
- Advanced Center of Electrical and Electronic Engineering, AC3E (ANID/FB0008)
- Arbotag Chile
This publication has 40 references indexed in Scilit:
- Urban street tree dataset for image classification and instance segmentationComputers and Electronics in Agriculture, 2023
- NDVI/NDRE prediction from standard RGB aerial imagery using deep learningComputers and Electronics in Agriculture, 2022
- An automatic approach for tree species detection and profile estimation of urban street trees using deep learning and Google street view imagesISPRS Journal of Photogrammetry and Remote Sensing, 2022
- Self-spectral learning with GAN based spectral–spatial target detection for hyperspectral imageNeural Networks, 2021
- Mapping trees along urban street networks with deep learning and street-level imageryISPRS Journal of Photogrammetry and Remote Sensing, 2021
- M2H-Net: A Reconstruction Method For Hyperspectral Remotely Sensed ImageryISPRS Journal of Photogrammetry and Remote Sensing, 2021
- Digital picture formats and representations2021
- From Google Maps to a fine-grained catalog of street treesISPRS Journal of Photogrammetry and Remote Sensing, 2018
- Near-Infrared Aerial Crop Mark Archaeology: From its Historical Use to Current Digital ImplementationsJournal of Archaeological Method and Theory, 2011
- Use of hyperspectral derivative ratios in the red-edge region to identify plant stress responses to gas leaksRemote Sensing of Environment, 2004