Deep Deterministic Policy Gradient Based on Double Network Prioritized Experience Replay
Open Access
- 20 April 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 9, 60296-60308
Abstract
The traditional deep deterministic policy gradient (DDPG) algorithm has the disadvantages of slow convergence velocity and ease of falling into the local optimum. From these two perspectives, a DDPG algorithm based on the double network prioritized experience replay mechanism (DNPER-DDPG) is proposed in this paper. Firstly, the value function is approximated by introducing the idea of two neural networks, and the minimum of the action value functions generated by the two networks is selected as the updated value of the actor policy network, which reduces the incidence of local optimal policy. Then, the Q values obtained by the two networks and the immediate reward obtained by the environment are used as the criteria for prioritization, and the importance of the samples in the experience replay mechanism is divided to improve the convergence speed of the algorithm. Finally, the improved method is demonstrated in the classic control environment of OpenAI Gym, and the results show that the proposed method has increased convergence speed and cumulative reward compared with the comparison algorithm.Keywords
Funding Information
- Key Funding Project of the Natural Science Foundation of China (61933007)
- Natural Science Foundation of China (61873058)
- Provincial Science Foundation of Heilongjiang (F2018004, F2018005)
This publication has 17 references indexed in Scilit:
- A novel DDPG method with prioritized experience replay2017
- Learning how to Active Learn: A Deep Reinforcement Learning Approach2017
- Memory trace replay: the shaping of memory consolidation by neuromodulationTrends in Neurosciences, 2015
- Human-level control through deep reinforcement learningNature, 2015
- The Arcade Learning Environment: An Evaluation Platform for General AgentsJournal of Artificial Intelligence Research, 2013
- Using continuous action spaces to solve discrete problems2009
- Reinforcement learning for robot soccerAutonomous Robots, 2009
- Prioritized sweeping: Reinforcement learning with less data and less timeMachine Learning, 1993
- Dynamic ProgrammingScience, 1966
- On the Theory of the Brownian MotionPhysical Review B, 1930