Xuguang Lan

Visual reinforcement learning (RL) relies on high-dimensional visual inputs but often suffers from low sample efficiency. While data augmentation enhances input diversity, it can inadvertently hinder policy learning by increasing visual complexity and reducing consistency. In this paper, we systematically analyze the efficiency bottlenecks associated with data augmentation and identify two key challenges: increased representational complexity and uncertainty in value estimation. To tackle these issues, we propose a novel framework, Feature Consistency and Self-value Distillation (FCSD). FCSD enforces feature consistency across augmented views and stabilizes value learning by distilling from its own past value network. We evaluate FCSD on several challenging RL benchmarks, including DMControl, ProcGen and Atari. Experimental results suggest that FCSD improves sample efficiency by mitigating augmentation-induced instability. Compared to several representative augmentation-based methods, FCSD demonstrates competitive or superior performance. Moreover, FCSD integrates seamlessly with both on-policy and off-policy RL algorithms, and demonstrates consistently robust and generalizable performance across diverse tasks and domains.

The increasing need for cooperation among intelligent machines has heightened the importance of cooperative multi-agent reinforcement learning (MARL). However, a dominant class of cooperative MARL approaches relies on monotonic value decomposition, which enables scalable decentralized execution but restricts the representable class of joint action-values. However, existing remedies bias learning targets toward high-value samples, which can be fragile under stochastic returns because optimistic emphasis may amplify lucky but suboptimal trajectories. To solve this challenge, we propose Target Transformation, which maps non-monotonic and stochastic learning targets into a monotonic-representable surrogate while preserving the optimal joint action. Building on this idea, we develop Uncertainty-aware Target Transformation (UT2) with value-based and policy-based instantiations that combine an uncertainty estimator with a best-individual coordination envelope. Experiments on diverse cooperative MARL benchmarks show that UT2 improves both performance and stability over strong baselines, with larger gains as non-monotonicity and stochasticity increase.

Model-free offline reinforcement learning learns policies from static, pre-collected datasets but often suffers from overestimating the values of out-of-distribution (OOD) data, which leads to biased value predictions and degraded performance. Existing methods mainly address this issue by penalizing OOD actions, but their extrapolation capabilities are limited by two key factors: (i) action-only penalties do not handle state-space extrapolation, causing cascading errors in unseen states; and (ii) uniform penalties suppress potentially beneficial extrapolation, biasing learning toward suboptimal dataset behavior. To overcome these limitations, we introduce ExtrapoBoost Offline Reinforcement Learning (EBORL), a method that improves extrapolation by jointly addressing both OOD states and actions. EBORL leverages an ensemble of Q-networks, each independently minimizing Temporal Difference error on in-distribution data, while maximizing the diversity of input gradients across the Q-networks for OOD state-action pairs. Theoretically, we provide a decomposition of the extrapolation error and prove that EBORL achieves a smaller upper bound on extrapolation error compared to baseline methods. Empirically, EBORL demonstrates strong performance on D4RL tasks with fewer ensemble heads and lower computational cost, and real-world evaluations further highlight its practical applicability.

Vision-based grasping, though widely employed for industrial and household applications, still struggles with object stacking scenarios. Current methods face three major challenges: limited inter-object relationship understanding; poor grasping adaptation across different viewpoints; and error propagation. To address the above challenges, we propose D2TriPO-DETR, a dual-decoder transformer with three outputs, of which are object detection, manipulation relationship, and grasp detection. Specifically, a distributed attention perception module and a rotation attention invariance module are designed to address limited interobject relationship understanding and poor grasping adaptation across different viewpoints. These two modules are respectively integrated into the two parallel decoders to output the triple results simultaneously, partly eliminating task-level error propagation. Experimental results on the visual manipulation relationship dataset indicate that D2TriPO-DETR outperforms existing state-of-the-art methods across all metrics, e.g., +6.1% object detection recall, +6.7% manipulation relationship image accuracy, and +1.5% grasp detection accuracy. Extensive real-world experiments and quantitative results validate D2TriPO-DETR’s effectiveness.

Vision-Language-Action (VLA) models are promising for generalist robot manipulation but remain brittle in out-of-distribution (OOD) settings, especially with limited real-robot data. To resolve the generalization bottleneck, we introduce a hierarchical Vision-Language-Action framework \our{} that leverages the generalization of large-scale pre-trained world model for robust and generalizable VIsual Subgoal TAsk decomposition VISTA. Our hierarchical framework \our{} consists of a world model as the high-level planner and a VLA as the low-level executor. The high-level world model first divides manipulation tasks into subtask sequences with goal images, and the low-level policy follows the textual and visual guidance to generate action sequences. Compared to raw textual goal specification, these synthesized goal images provide visually and physically grounded details for low-level policies, making it feasible to generalize across unseen objects and novel scenarios. We validate both visual goal synthesis and our hierarchical VLA policies in massive out-of-distribution scenarios, and the performance of the same-structured VLA in novel scenarios could boost from 14% to 69% with the guidance generated by the world model. Results demonstrate that our method outperforms previous baselines with a clear margin, particularly in out-of-distribution scenarios. Project page: \href{https://vista-wm.github.io/}{https://vista-wm.github.io}

In reinforcement learning (RL) based robot skill acquisition, a high-fidelity simulator is usually indispensable but unattainable since the real environment dynamics are difficult to model, which leads to severe sim-to-real gaps. Existing methods solve this problem by combining offline and online RL to jointly learn transferable policies from limited offline data and imperfect simulators. However, due to the unrestricted exploration in the imperfect simulator, the hybrid offline-and-online RL methods inevitably suffer from low sample efficiency and insufficient state-action space coverage during training. To solve this problem, we propose a State Revisit and Re-exploration (SR2) hybrid offline-and-online RL framework. In particular, the proposed algorithm employs a meta-policy and a sub-policy, where the meta-policy aims to find high-quality states in the offline trajectories for online exploration, and the sub-policy learns the robot skill using mixed offline and online data. By introducing the state revisit and explore mechanism, our approach efficiently improves performance on a set of sim-to-real robotic tasks. Through extensive simulation and real-world tasks, we demonstrate the superior performance of our approach against other state-of-the-art methods.