Penglei Sun
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Penglei Sun has a total of 62 co-authors. Here are the top co-authors by number of shared publications:
- Xiaowen Chu
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China
10 shared publications - Yaoxian Song
Zhejiang University, Hangzhou, China
6 shared publications - Qiang Wang
Harbin Institute of Technology (Shenzhen), Shenzhen, China
5 shared publications - Tiefeng Li
Center for X‐Mechanics Department of Engineering Mechanics Zhejiang University Hangzhou China
5 shared publications - Zhixu Li
School of Information, Renmin University of China, Beijing, China and School of Smart Governance, Renmin University of China, Beijing, China
5 shared publications - Xiangru Zhu
Fudan University, Shanghai, China
3 shared publications
Latest Publications
- Open Access
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 9500-9509
SpatialGrammar: A Domain-Specific Language for LLM-Based 3D Indoor Scene Generation
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 9500-9509
Open AccessSpatialGrammar: A Domain-Specific Language for LLM-Based 3D Indoor Scene Generation
- 1 October 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Internet of Things Journal
- Vol. 13 (19) , 44174-44195
From Ground to Space: A Survey on Multidomain Data Lifecycle for Urban Embodied Agents
- 1 October 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Internet of Things Journal
- Vol. 13 (19) , 44174-44195
From Ground to Space: A Survey on Multidomain Data Lifecycle for Urban Embodied Agents
- 10 September 2026
- Published by Springer Nature
- p. 465-483
Rule-VLN: Bridging Perception and Compliance via Semantic Reasoning and Geometric Rectification
- 10 September 2026
- Published by Springer Nature
- p. 465-483
Rule-VLN: Bridging Perception and Compliance via Semantic Reasoning and Geometric Rectification
- Open Access
- 11 March 2026
- Published by MDPI AG in Preprints.org
From Terrain to Space: A Survey on Multidomain Data Lifecycle for Urban Embodied Agents
Urban Embodied Agents (UrbanEAs) are emerging to actively interact with complex, large-scale city environments and generate vast, heterogeneous data streams, moving beyond the single-vehicle of existing autonomous driving. However, urban environments present distinct challenges, including environmental variability, limited observability, and interaction complexity. These challenges hinder the effectiveness of existing embodied agents, which have focused on controlled indoor environments, and expose the inherent limitations of relying on single-domain data. Therefore, establishing a comprehensive data lifecycle to fuse multidomain data from terrain, aerial, and space is a strategy for developing actionable embodied capabilities from raw urban streams. Distinct from existing surveys that follow a model-centric paradigm for urban computing or autonomous driving, we systematically propose and review a comprehensive Data Lifecycle from a multidomain data perspective, which is critical for the UrbanEA. First, we propose a unified framework containing four key stages of this lifecycle: Data Perception, Data Management, Data Modeling, and Task Application. Next, we establish a taxonomy for each stage of the lifecycle. Specifically, we detail the evolution from static data storage to active agent memory, and analyze integration strategies designed to bridge multidomain gaps. We demonstrate how UrbanEAs empower downstream tasks, including Urban Scene Question-Answering (SQA), Vision-Language Navigation (VLN), and Human-Agent Collaboration (HAC). Finally, we outline the social impact of the data lifecycle of UrbanEA and open research problems with the future directions. Our survey provides a roadmap for designing the robust, high-performance data frameworks essential for these UrbanEAs.
- 11 March 2026
- Published by MDPI AG in Preprints.org
Open AccessFrom Terrain to Space: A Survey on Multidomain Data Lifecycle for Urban Embodied Agents
Urban Embodied Agents (UrbanEAs) are emerging to actively interact with complex, large-scale city environments and generate vast, heterogeneous data streams, moving beyond the single-vehicle of existing autonomous driving. However, urban environments present distinct challenges, including environmental variability, limited observability, and interaction complexity. These challenges hinder the effectiveness of existing embodied agents, which have focused on controlled indoor environments, and expose the inherent limitations of relying on single-domain data. Therefore, establishing a comprehensive data lifecycle to fuse multidomain data from terrain, aerial, and space is a strategy for developing actionable embodied capabilities from raw urban streams. Distinct from existing surveys that follow a model-centric paradigm for urban computing or autonomous driving, we systematically propose and review a comprehensive Data Lifecycle from a multidomain data perspective, which is critical for the UrbanEA. First, we propose a unified framework containing four key stages of this lifecycle: Data Perception, Data Management, Data Modeling, and Task Application. Next, we establish a taxonomy for each stage of the lifecycle. Specifically, we detail the evolution from static data storage to active agent memory, and analyze integration strategies designed to bridge multidomain gaps. We demonstrate how UrbanEAs empower downstream tasks, including Urban Scene Question-Answering (SQA), Vision-Language Navigation (VLN), and Human-Agent Collaboration (HAC). Finally, we outline the social impact of the data lifecycle of UrbanEA and open research problems with the future directions. Our survey provides a roadmap for designing the robust, high-performance data frameworks essential for these UrbanEAs.
- Open Access
From Literature to Lab: Closed-Loop Advancement of Perovskite Solar Cells via Domain Knowledge Guided LLM
Perovskite solar cells (PSCs) have been considered as a next-generation disruptive photovoltaic technology, yet their advancement is constrained by the complexity of perovskite recipe with high-dimensional material and process design space. Despite the impressive general reasoning of Large Language Models (LLMs), they struggle with two limitations for application in PSCs: an inability to align general semantics with the perovskite domain knowledge, and an inefficiency in navigating high-dimensional perovskite material and recipe design spaces. To address these limitations, we introduce a domain-knowledge-guided framework PVK-LLM, a specialized model to serve as an expert to bridge general semantics with perovskite domain knowledge. By integrating this domain knowledge into a hierarchical Bayesian Optimization workflow, our approach efficiently navigates the high-dimension design space on a solar cell simulator platform. The domain knowledge resolves cold-start problems while dynamically adapting to simulator feedback. Moreover, in an individual wet-lab experiment aimed at maximizing power conversion efficiency (PCE), our framework autonomously proposes a novel synergistic four-component recipe comprising specialized organic passivation recipe (3MTPAI, PDAI2, EDAI2, and PipDI) which has not been reported in existing literature. This AI-designed recipe effectively achieves a champion PCE value of over 26.0 %, approaching world records achieved through extensive expert trial-and-error. Our approach can effectively enable LLM comprehend the domain knowledge, which can efficiently navigate in a high-dimensional, capable to accelerate the advancement in real-world perovskite as well as other material science development.
Open AccessFrom Literature to Lab: Closed-Loop Advancement of Perovskite Solar Cells via Domain Knowledge Guided LLM
Perovskite solar cells (PSCs) have been considered as a next-generation disruptive photovoltaic technology, yet their advancement is constrained by the complexity of perovskite recipe with high-dimensional material and process design space. Despite the impressive general reasoning of Large Language Models (LLMs), they struggle with two limitations for application in PSCs: an inability to align general semantics with the perovskite domain knowledge, and an inefficiency in navigating high-dimensional perovskite material and recipe design spaces. To address these limitations, we introduce a domain-knowledge-guided framework PVK-LLM, a specialized model to serve as an expert to bridge general semantics with perovskite domain knowledge. By integrating this domain knowledge into a hierarchical Bayesian Optimization workflow, our approach efficiently navigates the high-dimension design space on a solar cell simulator platform. The domain knowledge resolves cold-start problems while dynamically adapting to simulator feedback. Moreover, in an individual wet-lab experiment aimed at maximizing power conversion efficiency (PCE), our framework autonomously proposes a novel synergistic four-component recipe comprising specialized organic passivation recipe (3MTPAI, PDAI2, EDAI2, and PipDI) which has not been reported in existing literature. This AI-designed recipe effectively achieves a champion PCE value of over 26.0 %, approaching world records achieved through extensive expert trial-and-error. Our approach can effectively enable LLM comprehend the domain knowledge, which can efficiently navigate in a high-dimensional, capable to accelerate the advancement in real-world perovskite as well as other material science development.
- Open Access
- 27 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 3448-3457
City-VLM: Towards Multidomain Perception Scene Understanding via Multimodal Incomplete Learning
- 27 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 3448-3457
Open AccessCity-VLM: Towards Multidomain Perception Scene Understanding via Multimodal Incomplete Learning
- Open Access
City-VLM: Towards Multidomain Perception Scene Understanding via Multimodal Incomplete Learning
Scene understanding enables intelligent agents to interpret and comprehend their environment. While existing large vision-language models (LVLMs) for scene understanding have primarily focused on indoor household tasks, they face two significant limitations when applied to outdoor large-scale scene understanding. First, outdoor scenarios typically encompass larger-scale environments observed through various sensors from multiple viewpoints (e.g., bird view and terrestrial view), while existing indoor LVLMs mainly analyze single visual modalities within building-scale contexts from humanoid viewpoints. Second, existing LVLMs suffer from missing multidomain perception outdoor data and struggle to effectively integrate 2D and 3D visual information. To address the aforementioned limitations, we build the first multidomain perception outdoor scene understanding dataset, named \textbf{\underline{SVM-City}}, deriving from multi\textbf{\underline{S}}cale scenarios with multi\textbf{\underline{V}}iew and multi\textbf{\underline{M}}odal instruction tuning data. It contains $420$k images and $4, 811$M point clouds with $567$k question-answering pairs from vehicles, low-altitude drones, high-altitude aerial planes, and satellite. To effectively fuse the multimodal data in the absence of one modality, we introduce incomplete multimodal learning to model outdoor scene understanding and design the LVLM named \textbf{\underline{City-VLM}}. Multimodal fusion is realized by constructing a joint probabilistic distribution space rather than implementing directly explicit fusion operations (e.g., concatenation). Experimental results on three typical outdoor scene understanding tasks show City-VLM achieves $18.14 \%$ performance surpassing existing LVLMs in question-answering tasks averagely. Our method demonstrates pragmatic and generalization performance across multiple outdoor scenes.Open AccessCity-VLM: Towards Multidomain Perception Scene Understanding via Multimodal Incomplete Learning
Scene understanding enables intelligent agents to interpret and comprehend their environment. While existing large vision-language models (LVLMs) for scene understanding have primarily focused on indoor household tasks, they face two significant limitations when applied to outdoor large-scale scene understanding. First, outdoor scenarios typically encompass larger-scale environments observed through various sensors from multiple viewpoints (e.g., bird view and terrestrial view), while existing indoor LVLMs mainly analyze single visual modalities within building-scale contexts from humanoid viewpoints. Second, existing LVLMs suffer from missing multidomain perception outdoor data and struggle to effectively integrate 2D and 3D visual information. To address the aforementioned limitations, we build the first multidomain perception outdoor scene understanding dataset, named \textbf{\underline{SVM-City}}, deriving from multi\textbf{\underline{S}}cale scenarios with multi\textbf{\underline{V}}iew and multi\textbf{\underline{M}}odal instruction tuning data. It contains $420$k images and $4, 811$M point clouds with $567$k question-answering pairs from vehicles, low-altitude drones, high-altitude aerial planes, and satellite. To effectively fuse the multimodal data in the absence of one modality, we introduce incomplete multimodal learning to model outdoor scene understanding and design the LVLM named \textbf{\underline{City-VLM}}. Multimodal fusion is realized by constructing a joint probabilistic distribution space rather than implementing directly explicit fusion operations (e.g., concatenation). Experimental results on three typical outdoor scene understanding tasks show City-VLM achieves $18.14 \%$ performance surpassing existing LVLMs in question-answering tasks averagely. Our method demonstrates pragmatic and generalization performance across multiple outdoor scenes. - 2 May 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Automation Science and Engineering
- Vol. 22 (15455955) , 15200-15214
Learning 6-DoF Fine-Grained Grasp Detection Based on Part Affordance Grounding
Robotic grasping is a fundamental ability for a robot to interact with the environment. Current methods focus on how to obtain a stable and reliable grasping pose in object level, while little work has been studied on part (shape)-wise grasping which is related to fine-grained grasping and robotic affordance. Parts can be seen as atomic elements to compose an object, which contains rich semantic knowledge and a strong correlation with affordance. However, lacking a large part-wise 3D robotic dataset limits the development of part representation learning and downstream applications. In this paper, we propose a new large Language-guided SHape grAsPing datasEt (named LangSHAPE) to promote 3D part-level affordance and grasping ability learning. From the perspective of robotic cognition, we design a two-stage fine-grained robotic grasping framework (named LangPartGPD), including a novel 3D part language grounding model and a partaware grasp pose detection model, in which explicit language input from human or large language models (LLMs) could guide a robot to generate part-level 6-DoF grasping pose with textual explanation. Our method combines the advantages of humanrobot collaboration and LLMs’ planning ability using explicit language as a symbolic intermediate. To evaluate the effectiveness of our proposed method, we perform 3D part grounding and fine-grained grasp detection experiments on both simulation and physical robot settings, following language instructions across different degrees of textual complexity. Results show our method achieves competitive performance in 3D geometry fine-grained grounding, object affordance inference, and 3D part-aware grasping tasks. Our dataset and code are available on our project website https://sites.google.com/view/lang-shape.
- 2 May 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Automation Science and Engineering
- Vol. 22 (15455955) , 15200-15214
Learning 6-DoF Fine-Grained Grasp Detection Based on Part Affordance Grounding
Robotic grasping is a fundamental ability for a robot to interact with the environment. Current methods focus on how to obtain a stable and reliable grasping pose in object level, while little work has been studied on part (shape)-wise grasping which is related to fine-grained grasping and robotic affordance. Parts can be seen as atomic elements to compose an object, which contains rich semantic knowledge and a strong correlation with affordance. However, lacking a large part-wise 3D robotic dataset limits the development of part representation learning and downstream applications. In this paper, we propose a new large Language-guided SHape grAsPing datasEt (named LangSHAPE) to promote 3D part-level affordance and grasping ability learning. From the perspective of robotic cognition, we design a two-stage fine-grained robotic grasping framework (named LangPartGPD), including a novel 3D part language grounding model and a partaware grasp pose detection model, in which explicit language input from human or large language models (LLMs) could guide a robot to generate part-level 6-DoF grasping pose with textual explanation. Our method combines the advantages of humanrobot collaboration and LLMs’ planning ability using explicit language as a symbolic intermediate. To evaluate the effectiveness of our proposed method, we perform 3D part grounding and fine-grained grasp detection experiments on both simulation and physical robot settings, following language instructions across different degrees of textual complexity. Results show our method achieves competitive performance in 3D geometry fine-grained grounding, object affordance inference, and 3D part-aware grasping tasks. Our dataset and code are available on our project website https://sites.google.com/view/lang-shape.
- 31 October 2024
- Published by Springer Nature
- p. 387-404
Multi-task Domain Adaptation for Language Grounding with 3D Objects
- 31 October 2024
- Published by Springer Nature
- p. 387-404
Multi-task Domain Adaptation for Language Grounding with 3D Objects
- Open Access
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 2156-2165
3D Question Answering for City Scene Understanding
3D multimodal question answering (MQA) plays a crucial role in scene understanding by enabling intelligent agents to comprehend their surroundings in 3D environments. While existing research has primarily focused on indoor household tasks and outdoor roadside autonomous driving tasks, there has been limited exploration of city-level scene understanding tasks. Furthermore, existing research faces challenges in understanding city scenes, due to the absence of spatial semantic information and human-environment interaction information at the city level.To address these challenges, we investigate 3D MQA from both dataset and method perspectives. From the dataset perspective, we introduce a novel 3D MQA dataset named City-3DQA for city-level scene understanding, which is the first dataset to incorporate scene semantic and human-environment interactive tasks within the city. From the method perspective, we propose a Scene graph enhanced City-level Understanding method (Sg-CityU), which utilizes the scene graph to introduce the spatial semantic. A new benchmark is reported and our proposed Sg-CityU achieves accuracy of 63.94 % and 63.76 % in different settings of City-3DQA. Compared to indoor 3D MQA methods and zero-shot using advanced large language models (LLMs), Sg-CityU demonstrates state-of-the-art (SOTA) performance in robustness and generalization.
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 2156-2165
Open Access3D Question Answering for City Scene Understanding
3D multimodal question answering (MQA) plays a crucial role in scene understanding by enabling intelligent agents to comprehend their surroundings in 3D environments. While existing research has primarily focused on indoor household tasks and outdoor roadside autonomous driving tasks, there has been limited exploration of city-level scene understanding tasks. Furthermore, existing research faces challenges in understanding city scenes, due to the absence of spatial semantic information and human-environment interaction information at the city level.To address these challenges, we investigate 3D MQA from both dataset and method perspectives. From the dataset perspective, we introduce a novel 3D MQA dataset named City-3DQA for city-level scene understanding, which is the first dataset to incorporate scene semantic and human-environment interactive tasks within the city. From the method perspective, we propose a Scene graph enhanced City-level Understanding method (Sg-CityU), which utilizes the scene graph to introduce the spatial semantic. A new benchmark is reported and our proposed Sg-CityU achieves accuracy of 63.94 % and 63.76 % in different settings of City-3DQA. Compared to indoor 3D MQA methods and zero-shot using advanced large language models (LLMs), Sg-CityU demonstrates state-of-the-art (SOTA) performance in robustness and generalization.