Zhenliang Zhang

Current artificial intelligence evaluation often relies on static benchmarks and task-specific performance metrics, which are useful for system comparison but limited in assessing developmental progression, cognitive architecture, adaptive transfer, and real-world applicability. Building on developmental approaches to AGI testing, including our earlier conceptual proposal, this article develops an operational formulation of the General–Specialized–Applicable (GSA) framework for artificial general intelligence (AGI) evaluation. The framework organizes evidence into three non-interchangeable stages: the General stage, which assesses foundational capacities for open-ended generalization, value-oriented regulation, and autonomy; the Specialized stage, which evaluates stable domain-specific competence; and the Applicable stage, which examines whether such competence can be deployed safely and robustly in realistic environments. The article further introduces a dynamic task-generation pipeline and a provisional operational rubric for interpreting stage-specific evidence. An illustrative case study using household tasks in a simulated embodied environment illustrates how GSA can provide diagnostic information beyond aggregate benchmark scores by identifying capability gaps in current multimodal large language model (MLLM) agents. Rather than offering a final universal standard, the GSA framework provides an evaluation-oriented structure for connecting benchmark performance with architectural readiness, specialization, and deployment-level applicability.

Precisely manipulating an unseen articulated object is a common yet challenging task, primarily due to the ambiguity in predicting the affordance and the articulation model of the object. Current data-driven methods learn spurious correlations through passive observation, which limits their adaptability to new objects and precision in manipulation tasks. To achieve robust generalization and precise manipulation, we propose a novel Active Kinematic Modeling method (AKM), where robots actively interact with objects to overcome spurious correlations and achieve precise kinematic modeling. Our method is composed of three stages. First, hypothesis-driven exploration identifies contact points where interactions with the object trigger detectable relative motion between its parts. Next, unsupervised articulation modeling analyzes the recorded motion frames to construct an articulation model of the object. Finally, the model enables closed-loop manipulation via on-the-fly relocalization and replanning for precise execution. We validate AKM in simulation across a diverse benchmark of 116 articulated objects and demonstrate its generalization ability by deploying it to a real-world robotic system, consistently achieving sub-centimeter manipulation accuracy.

The integration of embodied agents into human environments demands embodied social intelligence: reasoning over both social norms and physical constraints. However, existing evaluations fail to address this integration, as they are limited to either disembodied social reasoning (e.g., in text) or socially-agnostic physical tasks. Both approaches fail to assess an agent's ability to integrate and trade off both physical and social constraints within a realistic, embodied context. To address this challenge, we introduce Spatially Situated Social Intelligence Test (S$^{3}$IT), a benchmark specifically designed to evaluate embodied social intelligence. It is centered on a novel and challenging seat-ordering task, requiring an agent to arrange seating in a 3D environment for a group of large language model-driven (LLM-driven) NPCs with diverse identities, preferences, and intricate interpersonal relationships. Our procedurally extensible framework generates a vast and diverse scenario space with controllable difficulty, compelling the agent to acquire preferences through active dialogue, perceive the environment via autonomous exploration, and perform multi-objective optimization within a complex constraint network. We evaluate state-of-the-art LLMs on S$^{3}$IT and found that they still struggle with this problem, showing an obvious gap compared with the human baseline. Results imply that LLMs have deficiencies in spatial intelligence, yet simultaneously demonstrate their ability to achieve near human-level competence in resolving conflicts that possess explicit textual cues.

In the pursuit of realizing artificial general intelligence (AGI), the importance of embodied artificial intelligence (AI) becomes increasingly apparent. Following this trend, research integrating robots with AGI has become prominent. As various kinds of embodiments have been designed, adaptability to diverse embodiments will become important to AGI. We introduce a new challenge, termed "Body Discovery of Embodied AI", focusing on tasks of recognizing embodiments and summarizing neural signal functionality. The challenge encompasses the precise definition of an AI body and the intricate task of identifying embodiments in dynamic environments, where conventional approaches often prove inadequate. To address these challenges, we apply causal inference method and evaluate it by developing a simulator tailored for testing algorithms with virtual environments. Finally, we validate the efficacy of our algorithms through empirical testing, demonstrating their robust performance in various scenarios based on virtual environments.

Despite recent advances in artificial intelligence (AI), it poses challenges to ensure personalized decision-making in tasks that are not considered in training datasets. To address this issue, we propose ValuePilot, a two-phase value-driven decision-making framework comprising a dataset generation toolkit DGT and a decision-making module DMM trained on the generated data. DGT is capable of generating scenarios based on value dimensions and closely mirroring real-world tasks, with automated filtering techniques and human curation to ensure the validity of the dataset. In the generated dataset, DMM learns to recognize the inherent values of scenarios, computes action feasibility and navigates the trade-offs between multiple value dimensions to make personalized decisions. Extensive experiments demonstrate that, given human value preferences, our DMM most closely aligns with human decisions, outperforming Claude-3.5-Sonnet, Gemini-2-flash, Llama-3.1-405b and GPT-4o. This research is a preliminary exploration of value-driven decision-making. We hope it will stimulate interest in value-driven decision-making and personalized decision-making within the community.

Demonstrative learning in virtual reality (VR) is a pivotal learning strategy for knowledge transfer for embodied agents. While existing studies have extensively explored agents’ knowledge transfer through self-demonstrative learning (SDL) or teacher-demonstrative learning (TDL), there has been limited focus on a system that integrates both paradigms. This article proposes a comprehensive system that combines the SDL paradigm with the TDL paradigm in VR from a top-down perspective. The system involves using directed probabilistic graphs (DPGs) for knowledge representation, constructing detectors and actuators based on object fluents and atomic actions, representing knowledge acquired from both learning paradigms on the DPGs, and incorporating knowledge integration and visualization. Through system evaluation, we show the advantages of integrating two demonstrative learning paradigms, including increased learning efficiency, mitigating demonstrator’s deficiencies, and more logical task execution. The study also reveals that a dynamically decreasing fusion factor with the learning progresses, performs well across different correct percentages of teacher demonstrations. Additionally, we show a more decentralized demonstration in the middle of the agent’s learning progress maximizing learning efficiency when demonstrating only a few atomic actions. Finally, we assess the system’s generalizability and delineate its boundaries. With the ongoing development and the increasing availability of human data in VR, we anticipate that our system can be applied to future scenarios of efficient human-agent knowledge transfer in human-agent symbiosis.

Virtual reality enables us to access and interact with immersive virtual environments anytime and anywhere in various fields such as entertainment, training, and education. However, users immersed in virtual scenes remain physically connected to their real-world surroundings, which can pose safety and immersion challenges. Although virtual scene synthesis has attracted widespread attention, many popular methods are limited to generating purely virtual scenes independent of physical environments or simply mapping physical objects as obstacles. To this end, we propose a scene agent that synthesizes situated 3D virtual scenes as a kind of ubiquitous embodied interface in VR for users. The scene agent synthesizes scenes by perceiving the user's physical environment as well as inferring the user's demands. The synthesized scenes maintain the affordances of the physical environment, enabling immersive users to interact with the physical environment and improving the user's sense of security. Meanwhile, the synthesized scenes maintain the style described by the user, improving the user's immersion. The comparison results show that the proposed scene agent can synthesize virtual scenes with better affordance maintenance, scene diversity, style maintenance, and 3D intersection over union compared to baselines. To the best of our knowledge, this is the first work that achieves in situ scene synthesis with virtual-real affordance consistency and user demand.

Human Action Recognition (HAR) is a very crucial task in computer vision. It helps to carry out a series of downstream tasks, like understanding human behaviors. Due to the complexity of human behaviors, many highly valuable behaviors are not yet encompassed within the available datasets for HAR, e.g., human habitual behaviors (HHBs). HHBs hold significant importance for analyzing a person's personality, habits, and psychological changes. To solve these problems, in this work, we build a novel video dataset to demonstrate various HHBs. These behaviors in the proposed dataset are able to reflect internal mental states and specific emotions of the characters, e.g., crossing arms suggests to shield oneself from perceived threats. The dataset contains 30 categories of habitual behaviors including more than 300,000 frames and 6,899 action instances. Since these behaviors usually appear at small local parts of human action videos, it is difficult for existing action recognition methods to handle these local features. Therefore, we also propose a two-stream model using both human skeletons and RGB appearances. Experimental results demonstrate that our proposed method has much better performance in action recognition than the existing methods on the proposed dataset.

We introduce Bongard-OpenWorld, a new benchmark for evaluating real-world few-shot reasoning for machine vision. It originates from the classical Bongard Problems (BPs): Given two sets of images (positive and negative), the model needs to identify the set that query images belong to by inducing the visual concepts, which is exclusively depicted by images from the positive set. Our benchmark inherits the few-shot concept induction of the original BPs while adding the two novel layers of challenge: 1) open-world free-form concepts, as the visual concepts in Bongard-OpenWorld are unique compositions of terms from an open vocabulary, ranging from object categories to abstract visual attributes and commonsense factual knowledge; 2) real-world images, as opposed to the synthetic diagrams used by many counterparts. In our exploration, Bongard-OpenWorld already imposes a significant challenge to current few-shot reasoning algorithms. We further investigate to which extent the recently introduced Large Language Models (LLMs) and Vision-Language Models (VLMs) can solve our task, by directly probing VLMs, and combining VLMs and LLMs in an interactive reasoning scheme. We even conceived a neuro-symbolic reasoning approach that reconciles LLMs & VLMs with logical reasoning to emulate the human problem-solving process for Bongard Problems. However, none of these approaches manage to close the human-machine gap, as the best learner achieves 64% accuracy while human participants easily reach 91%. We hope Bongard-OpenWorld can help us better understand the limitations of current visual intelligence and facilitate future research on visual agents with stronger few-shot visual reasoning capabilities.

Artificial intelligence (AI) has revolutionized human cognitive abilities and facilitated the development of new AI entities capable of interacting with humans in both physical and virtual environments. Despite the existence of virtual reality, mixed reality, and augmented reality for several years, integrating these technical fields remains a formidable challenge due to their disparate application directions. The advent of AI agents, capable of autonomous perception and action, further compounds this issue by exposing the limitations of traditional human-centered research approaches. It is imperative to establish a comprehensive framework that accommodates the dual perceptual centers of humans and AI agents in both physical and virtual worlds. In this paper, we introduce the symmetrical reality framework, which offers a unified representation encompassing various forms of physical-virtual amalgamations. This framework enables researchers to better comprehend how AI agents can collaborate with humans and how distinct technical pathways of physical-virtual integration can be consolidated from a broader perspective. We then delve into the coexistence of humans and AI, demonstrating a prototype system that exemplifies the operation of symmetrical reality systems for specific tasks, such as pouring water. Subsequently, we propose an instance of an AI-driven active assistance service that illustrates the potential applications of symmetrical reality. This paper aims to offer beneficial perspectives and guidance for researchers and practitioners in different fields, thus contributing to the ongoing research about human-AI coexistence in both physical and virtual environments.