Name
Title | ||
|---|---|---|
Updates of Equilibrium Prop Match Gradients of Backprop Through Time in an RNN with Static Input. |
Abstract | ||
|---|---|---|
Equilibrium Propagation (EP) is a biologically inspired learning algorithm for convergent recurrent neural networks, i.e. RNNs that are fed by a static input x and settle to a steady state. Training convergent RNNs consists in adjusting the weights until the steady state of output neurons coincides with a target y. Convergent RNNs can also be trained with the more conventional Backpropagation Through Time (BPTT) algorithm. In its original formulation EP was described in the case of real-time neuronal dynamics, which is computationally costly. In this work, we introduce a discrete -time version of EP with simplified equations and with reduced simulation time, bringing EP closer to practical machine learning tasks. We first prove theoretically, as well as numerically that the neural and weight updates of EP, computed by forward-time dynamics, are step-by-step equal to the ones obtained by BPTT, with gradients computed backward in time. The equality is strict when the transition function of the dynamics derives from a primitive function and the steady state is maintained long enough. We then show for more standard discrete -time neural network dynamics that the same property is approximately respected and we subsequently demonstrate training with EP with equivalent performance to BPTT. In particular, we define the first convolutional architecture trained with EP achieving similar to 1% test error on MNIST, which is the lowest error reported with ER. These results can guide the development of deep neural networks trained with EP. |
Year | Venue | Field |
|---|---|---|
2019 | ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 32 (NIPS 2019) | Backpropagation through time,Mathematical optimization,MNIST database,Recurrent neural network,Algorithm,Steady state,Artificial neural network,Transition function,Mathematics,Deep neural networks |
DocType | Volume | ISSN |
Journal | 32 | 1049-5258 |
Citations | PageRank | References |
0 | 0.34 | 0 |
Authors | ||
5 | ||
Name | Order | Citations | PageRank |
|---|---|---|---|
| Maxence Ernoult | 1 | 2 | 1.11 |
| Julie Grollier | 2 | 28 | 5.07 |
| Damien Querlioz | 3 | 279 | 27.77 |
| Yoshua Bengio | 4 | 42677 | 3039.83 |
| Benjamin Scellier | 5 | 27 | 2.08 |