Name
Abstract | ||
|---|---|---|
We consider the problem of developing rapid, stable, and scalable stochastic gradient descent algorithms for optimisation of very large nonlinear systems. Based on earlier work by Orr et al. on adaptive momentum--an efficient yet extremely unstable stochastic gradient descent algorithm--we develop a stabilised adaptive momentum algorithm that is suitable for noisy nonlinear optimisation problems. The stability is improved by introducing a forgetting factor 0 驴 驴 驴 1 that smoothes the trajectory and enables adaptation in non-stationary environments. The scalability of the new algorithm follows from the fact that at each iteration the multiplication by the curvature matrix can be achieved in O (n) steps using automatic differentiation tools. We illustrate the behaviour of the new algorithm on two examples: a linear neuron with squared loss and highly correlated inputs, and a multilayer perceptron applied to the four regions benchmark task. |
Year | Venue | Keywords |
|---|---|---|
2002 | ICANN | nonlinear systems,curvature matrix,new algorithm,unstable stochastic gradient descent,scalable stochastic gradient descent,stabilised adaptive momentum algorithm,correlated input,noisy nonlinear optimisation problem,automatic differentiation tool,large nonlinear system,rapid online learning,stable adaptive momentum,adaptive momentum,multilayer perceptron,nonlinear system,stochastic gradient descent |
DocType | Volume | ISSN |
Conference | 2415 | 0302-9743 |
ISBN | Citations | PageRank |
3-540-44074-7 | 3 | 0.88 |
References | Authors | |
6 | 2 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Thore Graepel | 1 | 3 | 0.88 |
| Nicol N. Schraudolph | 2 | 1185 | 164.26 |