Name
Title | ||
|---|---|---|
A tutorial on stochastic approximation algorithms for training Restricted Boltzmann Machines and Deep Belief Nets |
Abstract | ||
|---|---|---|
In this study, we provide a direct comparison of the Stochastic Maximum Likelihood algorithm and Contrastive Divergence for training Restricted Boltzmann Machines using the MNIST data set. We demonstrate that Stochastic Maximum Likelihood is superior when using the Restricted Boltzmann Machine as a classifier, and that the algorithm can be greatly improved using the technique of iterate averaging from the field of stochastic approximation. We further show that training with optimal parameters for classification does not necessarily lead to optimal results when Restricted Boltzmann Machines are stacked to form a Deep Belief Network. In our experiments we observe that fine tuning a Deep Belief Network significantly changes the distribution of the latent data, even though the parameter changes are negligible. |
Year | DOI | Venue |
|---|---|---|
2010 | 10.1109/ITA.2010.5454138 | Information Theory and Applications Workshop |
Keywords | Field | DocType |
approximation theory,belief networks,iterative methods,maximum likelihood estimation,stochastic processes,contrastive divergence,deep belief network,iterate averaging,maximum likelihood algorithm,restricted Boltzmann machines,stochastic approximation | Approximation algorithm,Restricted Boltzmann machine,Boltzmann machine,Mathematical optimization,Combinatorics,MNIST database,Computer science,Deep belief network,Algorithm,Approximation theory,Stochastic process,Stochastic approximation | Conference |
ISBN | Citations | PageRank |
978-1-4244-7014-3 | 29 | 1.49 |
References | Authors | |
22 | 4 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Kevin Swersky | 1 | 1118 | 52.13 |
| Bo Chen | 2 | 1097 | 33.93 |
| Benjamin Marlin | 3 | 950 | 95.15 |
| Nando De Freitas | 4 | 3284 | 273.68 |