StreetViewGPT: A Street View Image Scene Perception and Reasoning Method Based on Multimodal Large Language Model
Open Access
- 18 March 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
- Vol. 19 (19391404) , 10755-10769
Abstract
Multimodal large language models (MLLMs) have demonstrated promising potential in addressing real-world vision-language tasks. However, existing MLLMs exhibit relatively limited capabilities in understanding street view images, struggling to achieve multi-level semantic understanding and spatial reasoning for region-level fine-grained inference in street view images, which to some extent limits their potential in geospatial artificial intelligence (GeoAI). This study constructs an instruction fine-tuning dataset to guide MLLMs in learning spatial knowledge and scene features from street-view images, called SVGPT-Instruct. This large-scale dataset for street view understanding encompasses 430K instruction samples and 110K street view images. Ultimately, we propose the StreetViewGPT, an MLLM-based method for street view image scene perception and reasoning. The StreetViewGPT is capable of uniformly handling various street view image tasks, including time estimation (TE), weather recognition (WR), scene classification (SC), spatial distance estimation (SDE), and scene graph generation (SGG). The experimental results show that in the TE, WR, and SC tasks, the accuracy of StreetViewGPT improved by 1.33%, 21.68%, and 18.83% respectively compared to the suboptimal model. In the SDE task, the accuracy of predictions falling within the three error tolerance intervals of [75%, 125%], [50%, 150%], and [25%, 175%] reached 88.3%, 97.7%, and 99.5%, respectively, which improved by 78.2%, 78.7%, and 58.6%. In the SGG task, Recall and Mean Recall reached 16.09% and 6.84%, respectively, which improved by 12.49% and 5.4% compared to Qwen2-VL. It demonstrates that StreetViewGPT possesses multi-level semantic understanding and spatial reasoning capabilities, ranging from global semantic perception at the image level to fine-grained reasoning at the region level. The proposed method enhances the recognition and cognition capabilities for spatial scenes in street view images, offering positive implications for smart city perception and autonomous driving navigation research.Keywords
Funding Information
- National Natural Science Foundation of China (42171456)
- National Natural Science Foundation of China (42571549)
- Key Laboratory of Smart Earth (SYS-ZX06-2024-01)
This publication has 28 references indexed in Scilit:
- The All-Seeing Project V2: Towards General Relation Comprehension of the Open World2024
- Global Streetscapes — A comprehensive dataset of 10 million street-level images across 688 cities for urban science and analyticsISPRS Journal of Photogrammetry and Remote Sensing, 2024
- Knowledge and topology: A two layer spatially dependent graph neural networks to identify urban functions with time-series street view imageISPRS Journal of Photogrammetry and Remote Sensing, 2023
- Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation2018
- Visual Relationship Detection with Language Priors2016
- Deep Residual Learning for Image Recognition2016
- You Only Look Once: Unified, Real-Time Object Detection2016
- Image retrieval using scene graphs2015
- Going deeper with convolutions2015
- Are we ready for autonomous driving? The KITTI vision benchmark suite2012