Name
Title | ||
|---|---|---|
Linear-least-squares initialization of multilayer perceptrons through backpropagation of the desired response |
Abstract | ||
|---|---|---|
Training multilayer neural networks is typically carried out using descent techniques such as the gradient-based backpropagation (BP) of error or the quasi-Newton approaches including the Levenberg-Marquardt algorithm. This is basically due to the fact that there are no analytical methods to find the optimal weights, so iterative local or global optimization techniques are necessary. The success of iterative optimization procedures is strictly dependent on the initial conditions, therefore, in this paper, we devise a principled novel method of backpropagating the desired response through the layers of a multilayer perceptron (MLP), which enables us to accurately initialize these neural networks in the minimum mean-square-error sense, using the analytic linear least squares solution. The generated solution can be used as an initial condition to standard iterative optimization algorithms. However, simulations demonstrate that in most cases, the performance achieved through the proposed initialization scheme leaves little room for further improvement in the mean-square-error (MSE) over the training set. In addition, the performance of the network optimized with the proposed approach also generalizes well to testing data. A rigorous derivation of the initialization algorithm is presented and its high performance is verified with a number of benchmark training problems including chaotic time-series prediction, classification, and nonlinear system identification with MLPs. |
Year | DOI | Venue |
|---|---|---|
2005 | 10.1109/TNN.2004.841777 | IEEE Transactions on Neural Networks |
Keywords | Field | DocType |
neural network initialization.,index terms—approximate least-squares training of multilayer perceptrons mlps,minimum mean-square-error sense,multilayer perceptrons,training set,linear-least-squares initialization,backpropagation bp of desired response,global optimization technique,levenberg-marquardt algorithm,initialization algorithm,standard iterative optimization algorithm,iterative optimization procedure,high performance,initial condition,benchmark training problem,neural network,benchmark testing,mean square error,iterative methods,least square,levenberg marquardt,minimum mean square error,neural networks,backpropagation,nonlinear system identification,global optimization,multilayer perceptron | Mathematical optimization,Global optimization,Iterative method,Computer science,Multilayer perceptron,Artificial intelligence,Initialization,Backpropagation,Artificial neural network,Perceptron,Linear least squares,Machine learning | Journal |
Volume | Issue | ISSN |
16 | 2 | 1045-9227 |
Citations | PageRank | References |
22 | 2.37 | 23 |
Authors | ||
5 | ||
Name | Order | Citations | PageRank |
|---|---|---|---|
| Deniz Erdogmus | 1 | 375 | 36.16 |
| O. Fontenla-Romero | 2 | 34 | 3.16 |
| J. C. Principe | 3 | 658 | 46.92 |
| A. Alonso-Betanzos | 4 | 148 | 11.18 |
| Enrique Castillo | 5 | 555 | 59.86 |