Paper Info
Title
V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control
Abstract
Some of the most successful applications of deep reinforcement learning to challenging domains in discrete and continuous control have used policy gradient methods in the on-policy setting. However, policy gradients can suffer from large variance that may limit performance, and in practice require carefully tuned entropy regularization to prevent policy collapse. As an alternative to policy gradient algorithms, we introduce V-MPO, an on-policy adaptation of Maximum a Posteriori Policy Optimization (MPO) that performs policy iteration based on a learned state-value function. We show that V-MPO surpasses previously reported scores for both the Atari-57 and DMLab-30 benchmark suites in the multi-task setting, and does so reliably without importance weighting, entropy regularization, or population-based tuning of hyperparameters. On individual DMLab and Atari levels, the proposed algorithm can achieve scores that are substantially higher than has previously been reported. V-MPO is also applicable to problems with high-dimensional, continuous action spaces, which we demonstrate in the context of learning to control simulated humanoids with 22 degrees of freedom from full state observations and 56 degrees of freedom from pixel observations, as well as example OpenAI Gym tasks where V-MPO achieves substantially higher asymptotic scores than previously reported.
Year
Venue
Keywords
2020
ICLR
reinforcement learning, policy iteration, multi-task learning, continuous control
DocType
Citations 
PageRank 
Conference
0
0.34
References 
Authors
25
14
Name
Order
Citations
PageRank
H. Francis Song11055.14
Abbas Abdolmaleki24612.82
Jost Tobias Springenberg3112662.86
Aidan Clark400.34
Hubert Soyer51357.32
Jack Rae6758.77
Seb Noury700.68
Arun Ahuja8727.45
Siqi Liu9554.94
Dhruva Tirumala1001.01
Nicolas Heess11176294.77
Dan Belov12211.73
Martin Riedmiller135655366.29
Matthew M Botvinick1449425.34