Name
Title | ||
|---|---|---|
Least Squares Solutions of the HJB Equation With Neural Network Value-Function Approximators |
Abstract | ||
|---|---|---|
In this paper, we present an empirical study of iterative least squares minimization of the Hamilton-Jacobi-Bellman (HJB) residual with a neural network (NN) approximation of the value function. Although the nonlinearities in the optimal control problem and NN approximator preclude theoretical guarantees and raise concerns of numerical instabilities, we present two simple methods for promoting convergence, the effectiveness of which is presented in a series of experiments. The first method involves the gradual increase of the horizon time scale, with a corresponding gradual increase in value function complexity. The second method involves the assumption of stochastic dynamics which introduces a regularizing second derivative term to the HJB equation. A gradual reduction of this term provides further stabilization of the convergence. We demonstrate the solution of several problems, including the 4-D inverted-pendulum system with bounded control. Our approach requires no initial stabilizing policy or any restrictive assumptions on the plant or cost function, only knowledge of the plant dynamics. In the Appendix, we provide the equations for first- and second-order differential backpropagation. |
Year | DOI | Venue |
|---|---|---|
2007 | 10.1109/TNN.2007.899249 | IEEE Transactions on Neural Networks |
Keywords | Field | DocType |
control nonlinearities,function approximation,least squares approximations,minimisation,neurocontrollers,nonlinear control systems,optimal control,stability,stochastic systems,HJB equation,Hamilton-Jacobi-Bellman residual,bounded control,horizon time scale,inverted-pendulum system,iterative least squares minimization,neural network,nonlinearities,optimal control,stabilization,stochastic dynamics,value function complexity,value-function approximator,Differential neural networks (NNs),Hamilton–Jacoby–Bellman (HJB) equation,dynamic programming,feedforward neural networks,optimal control,viscosity solution | Least squares,Hamilton–Jacobi–Bellman equation,Differential equation,Feedforward neural network,Mathematical optimization,Optimal control,Function approximation,Iterative method,Bellman equation,Mathematics | Journal |
Volume | Issue | ISSN |
18 | 4 | 1045-9227 |
Citations | PageRank | References |
17 | 1.42 | 18 |
Authors | ||
2 | ||
Name | Order | Citations | PageRank |
|---|---|---|---|
| Yuval Tassa | 1 | 1097 | 52.33 |
| Tom Erez | 2 | 1027 | 50.56 |