UrbanLLaVA: A Multi-modal Large Language Model for Urban Intelligence with Spatial Reasoning and Understanding
Preprint
Abstract
Urban research involves a wide range of scenarios and tasks that require the understanding of multi-modal data. Current methods often focus on specific data types and lack a unified framework in urban field for processing them comprehensively. The recent success of multi-modal large language models (MLLMs) presents a promising opportunity to overcome this limitation. In this paper, we introduce $\textit{UrbanLLaVA}$, a multi-modal large language model designed to process these four types of data simultaneously and achieve strong performance across diverse urban tasks compared with general MLLMs. In $\textit{UrbanLLaVA}$, we first curate a diverse urban instruction dataset encompassing both single-modal and cross-modal urban data, spanning from location view to global view of urban environment. Additionally, we propose a multi-stage training framework that decouples spatial reasoning enhancement from domain knowledge learning, thereby improving the compatibility and downstream performance of $\textit{UrbanLLaVA}$ across diverse urban tasks. Finally, we also extend existing benchmark for urban research to assess the performance of MLLMs across a wide range of urban tasks. Experimental results from three cities demonstrate that $\textit{UrbanLLaVA}$ outperforms open-source and proprietary MLLMs in both single-modal tasks and complex cross-modal tasks and shows robust generalization abilities across cities. Source codes and data are openly accessible to the research community via https://github.com/tsinghua-fib-lab/UrbanLLaVA.
Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- Toward large reasoning models: A survey of reinforced reasoning with large language modelsPatterns, 2025
- RSGPT: A remote sensing vision language model and benchmarkISPRS Journal of Photogrammetry and Remote Sensing, 2025
- SkyEyeGPT: Unifying remote sensing vision-language tasks via instruction tuning with large language modelISPRS Journal of Photogrammetry and Remote Sensing, 2025
- Dolphins: Multimodal Language Model for Driving2024
- ShareGPT4V: Improving Large Multi-modal Models with Better Captions2024
- LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Model2024
- Deep learning for cross-domain data fusion in urban computing: Taxonomy, advances, and outlookInformation Fusion, 2024
- EarthGPT: A Universal Multimodal Large Language Model for Multisensor Image Comprehension in Remote Sensing DomainIEEE Transactions on Geoscience and Remote Sensing, 2024
- Urban visual intelligence: Uncovering hidden city profiles with street view imagesProceedings of the National Academy of Sciences, 2023
- Street view imagery in urban analytics and GIS: A reviewLandscape and Urban Planning, 2021