Zhongyu Wei

A large language model (LLM) agent is a model coupled to a harness: parametric weights that hold knowledge and reasoning ability, wrapped in an external apparatus of memory, tools, and orchestration through which the model perceives and acts. Progress over the past years has come from pushing on both, with stronger models and richer harnesses, but the design decision that quietly governs an agent’s behavior is where competence should live: which knowledge, skills, and control to internalize into the model’s parameters, and which to externalize onto the surrounding harness. These decisions increasingly determine whether an agent is reliable, efficient, and able to keep improving. We investigate this core problem through the lens of the Theory of Agent (ToA), which views an LLM agent not as a task executor but as an epistemic decision-maker in terms of reasoning or acting at every step. The two means it has for that epistemic uncertainty reduction, namely reasoning over what it already holds and acting on the world to obtain what it does not, are the run-time faces of internalization and externalization: reasoning draws on competence that has been absorbed into the model, while acting exploits competence that remains in the harness. We review this internalization and externalization across five chapters. Foundations sets up the basic picture: an agent that, at each step, decides whether to answer from what it already knows or reach outside for what it does not, and what separates a good version of that decision from a wasteful one. Learning examines how training places competence on either side of the boundary, spanning internalization of memory, reasoning, tool-use policy, and world models, and externalization of memory, skills, and orchestration. Aligning treats calibration as the connecting thread across an agent’s cognitive boundaries, efficiency constraints, human preferences, environment constraints, and other agents. Evolving studies how agents revise both sides of the boundary over time, from model-side and harness-side updates to agent–environment co-evolution. Evaluation reinterprets metrics and benchmarks by the epistemic effort they expose and argues for measuring the boundary decision rather than task completion alone. We conclude with the open problems that follow once internalization and externalization are treated as the governing axis of agent design.

Large Language Models (LLMs), with remarkable conversational capabilities, have emerged as AI assistants that can handle both visual and textual modalities. However, their effectiveness in joint video-language understanding has not been extensively explored. In the article, we introduce Valley , a multi-modal foundation model designed to enable enhanced video comprehension and instruction-following capabilities. To this end, we construct two datasets, namely “ Valley-702k ” and “ Valley-instruct-73k ,” to cover a diverse range of video–text alignment and video-based instruction tasks, such as multi-shot captions, long video descriptions, action recognition, causal inference, and so on. Then, we adopt ViT-L/14 as the vision encoder and explore three different temporal modeling modules to learn multifaceted features for enhanced video understanding. In addition, we implement a two-phase training approach for Valley: the first phase focuses solely on training the projection module to facilitate the LLM’s capacity to understand visual input, and the second phase jointly trains the projection module and the LLM to improve their instruction following ability. Extensive experiments demonstrate that Valley has the potential to serve as an effective video assistant, simplifying complex video understanding tasks. Our code and data are publicly available at https://github.com/RupertLuo/Valley .

Traditional sociological research often relies on human participation, which, though effective, is expensive, challenging to scale, and with ethical concerns. Recent advancements in large language models (LLMs) highlight their potential to simulate human behavior, enabling the replication of individual responses and facilitating studies on many interdisciplinary studies. In this article, we conduct a comprehensive survey of this field, illustrating the recent progress in simulation driven by LLM-empowered agents. We categorize the simulations into three types: (1) Individual Simulation , which mimics specific individuals or demographic groups; (2) Scenario Simulation , where multiple agents collaborate to achieve goals within specific contexts; and (3) Society Simulation , which models interactions within agent societies to reflect the complexity and variety of real-world dynamics. These simulations follow a progression, ranging from detailed individual modeling to large-scale societal phenomena. We provide a detailed discussion of each simulation type, including the architecture or key components of the simulation, the classification of objectives or scenarios, and the evaluation method. Afterward, we summarize commonly used datasets, benchmarks and tools. Finally, we discuss the risks and challenges across these three types of simulation. A repository for the related sources is at https://github.com/FudanDISC/SocialAgent .

User modeling characterizes individuals through their preferences and behavioral patterns to enable personalized simulation and generation with Large Language Models (LLMs) in contemporary approaches. However, existing methods, whether prompt-based or training-based methods, face challenges in balancing personalization quality against computational and data efficiency. We propose a novel framework CURP, which employs a bidirectional user encoder and a discrete prototype codebook to extract multi-dimensional user traits. This design enables plug-and-play personalization with a small number of trainable parameters (about 20M parameters, about 0.2\% of the total model size). Through extensive experiments on variant generation tasks, we show that CURP achieves superior performance and generalization compared to strong baselines, while offering better interpretability and scalability. The code are available at https://github.com/RaidonWong/CURP_code

Interactive medical consultation requires an agent to proactively elicit missing clinical evidence under uncertainty. Yet existing evaluations largely remain static or outcome-centric, neglecting the evidence-gathering process. In this work, we propose an interactive evaluation framework that explicitly models the consultation process using a simulated patient and a \rev{simulated reporter} grounded in atomic evidences. Based on this representation, we introduce Information Coverage Rate (ICR) to quantify how completely an agent uncovers necessary evidence during interaction. To support systematic study, we build EviMed, an evidence-based benchmark spanning diverse conditions from common complaints to rare diseases, and evaluate 10 models with varying reasoning abilities. We find that strong diagnostic reasoning does not guarantee effective information collection, and this insufficiency acts as a primary bottleneck limiting performance in interactive settings. To address this, we propose REFINE, a strategy that leverages diagnostic verification to guide the agent in proactively resolving uncertainties. Extensive experiments demonstrate that REFINE consistently outperforms baselines across diverse datasets and facilitates effective model collaboration, enabling smaller agents to achieve superior performance under strong reasoning supervision. Our code can be found at https://github.com/NanshineLoong/EID-Benchmark .

Large language models (LLMs) alignment ensures model behaviors reflect human value. Existing alignment strategies primarily follow two paths: one assumes a universal value set for a unified goal (i.e., one-size-fits-all), while the other treats every individual as unique to customize models (i.e., individual-level). However, assuming a monolithic value space marginalizes minority norms, while tailoring individual models is prohibitively expensive. Recognizing that human society is organized into social clusters with high intra-group value alignment, we propose community-level alignment as a "middle ground". Practically, we introduce CommunityBench, the first large-scale benchmark for community-level alignment evaluation, featuring four tasks grounded in Common Identity and Common Bond theory. With CommunityBench, we conduct a comprehensive evaluation of various foundation models on CommunityBench, revealing that current LLMs exhibit limited capacity to model community-specific preferences. Furthermore, we investigate the potential of community-level alignment in facilitating individual modeling, providing a promising direction for scalable and pluralistic alignment.