Curiosity-Driven Distributional Soft Actor-Critic for AUV Anti-Disturbance Path Tracking
- 25 August 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Transportation Systems
- Vol. 26 (10) , 15568-15581
Abstract
With the advancement of marine resource exploration and exploitation technologies, autonomous underwater vehicles (AUVs) have shown significant potential to perform underwater tasks, such as pipeline maintenance. However, traditional control algorithms, such as proportional-integral-derivative, sliding mode, and model predictive control, struggle to adapt to nonlinear and current-disturbed underwater environments, which impedes their accuracy and stability in path-tracking tasks. To address these challenges, this paper proposes a curiosity-driven distributional soft actor-critic framework. The framework leverages distributional soft actor-critic algorithms to control the navigation direction of the AUV and employs traditional proportional-integral control to maintain navigation speed, creating a stable, high-precision control strategy for complex underwater environments. Building on this framework, this paper further advances its capabilities through two key improvements. First, a curiosity-driven automatic entropy adjustment technique is designed to enhance the exploration of unknown states and the utilization of similar states, thus optimizing the accuracy of path tracking. Second, a dynamic prioritized experience replay mechanism is developed to prioritize learning samples based on time-differential errors and historical rewards, thereby improving learning efficiency and stability. Simulation experiments are conducted in two different underwater environments with and without ocean currents. Compared with reinforcement learning-based methods and traditional control algorithms, the proposed method has significant advantages in terms of accuracy, stability, and adaptability.Keywords
Funding Information
- National Natural Science Foundation of China (U22A2011)
- National Natural Science Foundation of China (62072072)
- Fundamental Research Funds for the Central Universities (N25LPY014)
This publication has 33 references indexed in Scilit:
- Relay Hindsight Experience Replay: Self-guided continual reinforcement learning for sequential object manipulation tasks with sparse rewardsNeurocomputing, 2023
- HCS-R-HER: Hierarchical reinforcement learning based on cross subtasks rainbow hindsight experience replayJournal of Computational Science, 2023
- Sampled-data control through model-free reinforcement learning with effective experience replayJournal of Automation and Intelligence, 2023
- Hybrid attention-oriented experience replay for deep reinforcement learning and its application to a multi-robot cooperative hunting problemNeurocomputing, 2022
- Robust yaw control of autonomous underwater vehicle based on fractional-order PID controllerOcean Engineering, 2022
- A novel path following approach for autonomous ships based on fast marching method and deep reinforcement learningOcean Engineering, 2022
- Deep reinforcement learning based trading agents: Risk curiosity driven learning for financial rules-based policyExpert Systems with Applications, 2021
- Random curiosity-driven exploration in deep reinforcement learningNeurocomputing, 2020
- Fast and slow curiosity for high-level exploration in reinforcement learningApplied Intelligence, 2020
- Deep reinforcement learning-based controller for path following of an unmanned surface vehicleOcean Engineering, 2019