Tianyi Zhang

The latest research indicates that Large Vision-Language Models (VLMs) have a wide range of applications in the field of remote sensing. However, the vast amount of image data in this field presents a challenge in selecting high-quality multimodal data, which are essential for saving computational resources and time. Therefore, we propose an adaptive fine-tuning algorithm for multimodal large models. The core steps of this algorithm involve two stages of data truncation. First, the vast dataset is projected into a semantic vector space, where the MiniBatchKMeans algorithm is used for automated clustering. This classification ensures that the data within each cluster exhibit high semantic similarity. Next, the data within each cluster are processed by calculating the translational difference between the original and perturbed data in the multimodal large model’s vector space. This difference serves as a generalization metric for the data. Based on this metric, we select data with high generalization potential for training. We apply this algorithm to train the InternLM-XComposer2-VL-7B model on two 3090 GPUs, using one-third of the GeoChat multimodal remote sensing dataset. The results demonstrate that our algorithm outperforms state-of-the-art baselines. The model trained on our optimally chosen one-third dataset, as validated through experiments, showed only a 1% reduction in performance across various remote sensing metrics compared to the model trained on the full dataset. This approach significantly preserved general-purpose capabilities while reducing training time by 68.2%. Furthermore, the model achieved scores of 89.86 and 77.19 on the UCMerced and AID evaluation datasets, respectively, surpassing the GeoChat dataset by 5.43 and 5.16 points. It only showed a 0.91-point average decrease on the LRBEN evaluation dataset.

Automated water body (WB) extraction is one of the hot research topics in the field of remote sensing image processing. To address the challenges of over-extraction and incomplete extraction in complex water scenes, we propose an encoder–decoder architecture semantic segmentation network for high-precision extraction of WBs called EDWNet. We integrate the Cross-layer Feature Fusion (CFF) module to solve difficulties in segmentation of WB edges, utilizing the Global Attention Mechanism (GAM) module to reduce information diffusion, and combining with the Deep Attention Module (DAM) module to enhance the model’s global perception ability and refine WB features. Additionally, an auxiliary head is incorporated to optimize the model’s learning process. In addition, we analyze the feature importance of bands 2 to 7 in Landsat 8 OLI images, constructing a band combination (RGB 763) suitable for algorithm’s WB extraction. When we compare EDWNet with various other semantic segmentation networks, the results on the test dataset show that EDWNet has the highest accuracy. EDWNet is applied to accurately extract WBs in the Weihe River basin from 2013 to 2021, and we quantitatively analyzed the area changes of the WBs during this period and their causes. The results show that EDWNet is suitable for WB extraction in complex scenes and demonstrates great potential in long time-series and large-scale WB extraction.

This paper introduces WaterGPT, a language model designed for complex multimodal tasks in hydrology. WaterGPT is applied in three main areas: (1) processing and analyzing data such as images and text in water resources, (2) supporting intelligent decision-making for hydrological tasks, and (3) enabling interdisciplinary information integration and knowledge-based Q&A. The model has achieved promising results. One core aspect of WaterGPT involves the meticulous segmentation of training data for the supervised fine-tuning phase, sourced from real-world data and annotated with high quality using both manual methods and GPT-series model annotations. These data are carefully categorized into four types: knowledge-based, task-oriented, negative samples, and multi-turn dialogues. Additionally, another key component is the development of a multi-agent framework called Water_Agent, which enables WaterGPT to intelligently invoke various tools to solve complex tasks in the field of water resources. This framework handles multimodal data, including text and images, allowing for deep understanding and analysis of complex hydrological environments. Based on this framework, WaterGPT has achieved over a 90% success rate in tasks such as object detection and waterbody extraction. For the waterbody extraction task, using Dice and mIoU metrics, WaterGPT’s performance on high-resolution images from 2013 to 2022 has remained stable, with accuracy exceeding 90%. Moreover, we have constructed a high-quality water resources evaluation dataset, EvalWater, which covers 21 categories and approximately 10,000 questions. Using this dataset, WaterGPT achieved the highest accuracy to date in the field of water resources, reaching 83.09%, which is about 17.83 points higher than GPT-4.

In recent years, with the rapid development of large language models, serval models such as GPT-4o have demonstrated extraordinary capabilities, surpassing human performance in various language tasks. As a result, many researchers have begun exploring their potential applications in the field of public opinion analysis. This study proposes a novel large-language-models-based method for public opinion event heat level prediction. First, we preprocessed and classified 62,836 Chinese hot event data collected between July 2022 and December 2023. Then, based on each event's online dissemination heat index, we used the MiniBatchKMeans algorithm to automatically cluster the events and categorize them into four heat levels (ranging from low heat to very high heat). Next, we randomly selected 250 events from each heat level, totalling 1,000 events, to build the evaluation dataset. During the evaluation process, we employed various large language models to assess their accuracy in predicting event heat levels in two scenarios: without reference cases and with similar case references. The results showed that GPT-4o and DeepseekV2 performed the best in the latter case, achieving prediction accuracies of 41.4% and 41.5%, respectively. Although the overall prediction accuracy remains relatively low, it is worth noting that for low-heat (Level 1) events, the prediction accuracies of these two models reached 73.6% and 70.4%, respectively. Additionally, the prediction accuracy showed a downward trend from Level 1 to Level 4, which correlates with the uneven distribution of data across the heat levels in the actual dataset. This suggests that with the more robust dataset, public opinion event heat level prediction based on large language models will have significant research potential for the future.

In the era of big data, making full use of remote sensing images to automatically extract surface water bodies (WBs) in complex environments is extremely challenging. Due to the weak capability of existing algorithms in extracting small WBs and WB edge information from remote sensing images, we proposed a new method—Multiscale Fusion SegFormer (MF-SegFormer)—for WB extraction in the Weihe River Basin of China using Landsat 8 OLI images. The MF-SegFormer method adopts a cascading approach to fuse features output by the SegFormer encoder at multiple scales. A feature fusion (FF) module is proposed to enhance the extraction of WB edge information, while an Atrous Spatial Pyramid Pooling (ASPP) module is employed to enhance the extraction of small WBs. Furthermore, we analyzed the impact of four kinds of band combinations on WB extraction by the MF-SegFormer model, including true color composite images, false color images, true color images, and false color images enhanced by Gaussian stretch. We also compared our proposed method with several different approaches. The results suggested that false color composite images enhanced by Gaussian stretching are beneficial for extracting WBs, and the MF-SegFormer model achieves the highest accuracy across the study area with a precision of 77.6%, recall of 84.4%, F1-score of 80.9%, and mean intersection over union (mIoU) of 83.9%. In addition, we used the determination coefficient (R²) and root-mean-square error (RMSE) to evaluate the performance of river width extraction. Our extraction results in an overall R² of 0.946 and an RMSE of 28.21 m for the mainstream width in the “Xi’an-Xianyang” section of the Weihe River. The proposed MF-SegFormer method used in this study outperformed other methods and was found to be more robust for WB extraction.