Qi Liu

Achieving reliable object manipulation while traversing complex terrains is the missing link between agile quadruped locomotion and practical autonomy. Specifically, using traditional end-to-end reinforcement learning (RL) for dynamic ball manipulation in rugged environments presents two key challenges. The first is coordinating distinct motion modalities to integrate terrain traversal and ball control seamlessly. The second is overcoming sparse rewards in end-to-end RL, which impedes efficient policy convergence. To address these challenges, we propose a hierarchical RL framework. A high-level policy, informed by proprioceptive data and ball position, adaptively switches between pre-trained low-level skills such as ball dribbling and rough terrain navigation. We further propose Dynamic Skill-Focused Policy Optimization to suppress gradients from inactive skills and enhance critical skill learning. Both simulation and real-world experiments validate that our method outperforms baseline approaches in dynamic ball manipulation across rugged terrains, highlighting its effectiveness in challenging environments.

Offline reinforcement learning (RL) aims to learn a policy that maximizes the expected return using a given static dataset of transitions. However, offline RL faces the distribution shift problem. The policy constraint offline RL method is proposed to solve the distribution shift problem. During the policy constraint offline RL training, it is important to ensure the difference between the learned policy and behavior policy within a given threshold. Thus, the learned policy heavily relies on the quality of the behavior policy. However, a problem exists in existing policy constraint methods: if the dataset contains many low-reward transitions, the learned will be contained with a suboptimal reference policy, leading to slow learning speed, low sample efficiency, and inferior performances. This paper shows that the sampling method in policy constraint offline RL that uses all the transitions in the dataset can be improved. A simple but efficient sample filtering method is proposed to improve the sample efficiency and the final performance. First, we evaluate the score of the transitions by average reward and average discounted reward of episodes in the dataset and extract the transition samples of high scores. Second, the high-score transition samples are used to train the offline RL algorithms. We verify the proposed method in a series of offline RL algorithms and benchmark tasks. Experimental results show that the proposed method outperforms baselines.

The standard paradigm of reinforcement learning (RL) is the Markov Decision Process (MDP) in which an agent learns to maximize the cumulative discounted rewards. The reward function in MDP is generally defined as the sum of multiple reward components, each designed to encapsulate a specific aspect of the expected policy. The discount factor $\gamma \in [0,1)$ decreases the future reward in the present value, which determines the effective time horizon for the agent. In the conventional MDP, all reward components are subject to the same discount factor regardless of their specific meanings. Although this convenient configuration simplifies the problem in the algorithm deployment, it sacrifices precision in defining the optimization problem and results in a temporal mismatch of rewards with diverse physical meanings. This paper proposes multi-discounting MDP (MDMDP), a novel model based on reward decomposition to solve the above problems. MDMDP allows practitioners to set separate discount factors for different reward components. This capability provides great flexibility in combining reward components at different timescales. Furthermore, this paper proposes an RL algorithm, multi-discounting Q-learning, to solve finite MDMDP. Moreover, we extend it to the deep RL version, including multi-discounting DQN for discrete action space tasks and multi-discounting actor-critic for continuous action space tasks. Experimental results demonstrate that the proposed methods improve flexibility and precision in modeling complex tasks, enhancing the alignment of the agent's policy with desired objectives.

Advancing the dynamic loco-manipulation capabilities of quadruped robots in complex terrains is crucial for performing diverse tasks. Specifically, dynamic ball manipulation in rugged environments presents two key challenges. The first is coordinating distinct motion modalities to integrate terrain traversal and ball control seamlessly. The second is overcoming sparse rewards in end-to-end deep reinforcement learning, which impedes efficient policy convergence. To address these challenges, we propose a hierarchical reinforcement learning framework. A high-level policy, informed by proprioceptive data and ball position, adaptively switches between pre-trained low-level skills such as ball dribbling and rough terrain navigation. We further propose Dynamic Skill-Focused Policy Optimization to suppress gradients from inactive skills and enhance critical skill learning. Both simulation and real-world experiments validate that our methods outperform baseline approaches in dynamic ball manipulation across rugged terrains, highlighting its effectiveness in challenging environments. Videos are on our website: dribble-hrl.github.io.