Paper Info
Title
An efficient L2-norm regularized least-squares temporal difference learning algorithm
Abstract
In reinforcement learning, when samples are limited in some real applications, Least-Squares Temporal Difference (LSTD) learning is prone to over-fitting, which can be overcome by the introduction of regularization. However, the solution of LSTD with regularization still depends on costly matrix inversion operations. In this paper we investigate the L2-norm regularized LSTD learning and propose an efficient algorithm to avoid expensive computational cost. We derive LSTD using Bellman operator along with projection operator. The L2-norm penalty is introduced to avoid over-fitting. We also describe the difference between Bellman residual minimization and LSTD. Then we propose an efficient recursive least-squares algorithm for L2-norm regularized LSTD, which can eliminate matrix inversion operations and decrease computational complexity effectively. We present empirical comparisons on the Boyan chain problem. The results show that the performance of the new algorithm is better than that of regularized LSTD.
Year
DOI
Venue
2013
10.1016/j.knosys.2013.02.010
Knowl.-Based Syst.
Keywords
Field
DocType
bellman residual minimization,l2-norm penalty,regularized lstd,bellman operator,temporal difference,new algorithm,l2-norm regularized least-squares,reinforcement learning,l2-norm regularized lstd learning,computational complexity,efficient recursive least-squares algorithm,efficient algorithm,recursive least squares,regularization
Computer science,Projection (linear algebra),Minification,Regularization (mathematics),Artificial intelligence,Reinforcement learning,Mathematical optimization,Temporal difference learning,Algorithm,Norm (mathematics),Machine learning,Recursive least squares filter,Computational complexity theory
Journal
Volume
ISSN
Citations 
45,
0950-7051
6
PageRank 
References 
Authors
0.41
19
3
Name
Order
Citations
PageRank
Shenglei Chen1184.05
Geng Chen290.78
Ruijun Gu390.91