Balancing State Exploration and Skill Diversity in Unsupervised Skill Discovery
- 26 March 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Cybernetics
- Vol. 55 (5) , 2234-2247
Abstract
Unsupervised skill discovery seeks to acquire different useful skills without extrinsic reward via unsupervised reinforcement learning (RL), with the discovered skills efficiently adapting to multiple downstream tasks in various ways. However, recent advanced skill discovery methods struggle to well balance state exploration and skill diversity, particularly when the potential skills are rich and hard to discern. In this article, we propose contrastive dynamic skill discovery (ComSD) which generates diverse and exploratory unsupervised skills through a novel intrinsic incentive, named contrastive dynamic reward. It contains a particle-based exploration reward to make agents access far-reaching states for exploratory skill acquisition, and a novel contrastive diversity reward to promote the discriminability between different skills. Moreover, a novel dynamic weighting mechanism between the above two rewards is proposed to balance state exploration and skill diversity, which further enhances the quality of the discovered skills. Extensive experiments and analysis demonstrate that ComSD can generate diverse behaviors at different exploratory levels for multijoint robots, enabling state-of-the-art adaptation performance on challenging downstream tasks. It can also discover distinguishable and far-reaching exploration skills in the challenging tree-like 2-D maze.Keywords
Funding Information
- National Natural Science Foundation of China (62136008, 62293545)
- International Partnership Program of the Chinese Academy of Sciences (104GJHZ2022013GC)
This publication has 16 references indexed in Scilit:
- A Survey of Imitation Learning: Algorithms, Recent Developments, and ChallengesIEEE Transactions on Cybernetics, 2024
- Efficient Reward Shaping for Multiagent SystemsIEEE Transactions on Control of Network Systems, 2024
- Offline Reinforcement Learning With Behavior Value RegularizationIEEE Transactions on Cybernetics, 2024
- Refine to the essence: Less-redundant skill learning via diversity clusteringEngineering Applications of Artificial Intelligence, 2024
- Data-Efficient Reinforcement Learning for Complex Nonlinear SystemsIEEE Transactions on Cybernetics, 2023
- Implicit Posteriori Parameter Distribution Optimization in Reinforcement LearningIEEE Transactions on Cybernetics, 2023
- Human-Level Control Through Directly Trained Deep Spiking Q-NetworksIEEE Transactions on Cybernetics, 2022
- Empirical Policy Optimization for n-Player Markov GamesIEEE Transactions on Cybernetics, 2022
- Human-level control through deep reinforcement learningNature, 2015
- Nearest Neighbor Estimates of EntropyAmerican Journal of Mathematical and Management Sciences, 2003