Name
Abstract | ||
|---|---|---|
The Restricted Boltzmann Machine (RBM), a special case of general Boltzmann Machines and a typical Probabilistic Graphical Models, has attracted much attention in recent years due to its powerful ability in extracting features and representing the distribution underlying the training data. A most commonly used algorithm in learning RBMs is called Contrastive Divergence (CD) proposed by Hinton, which starts a Markov chain at a data point and runs the chain for only a few iterations to get a low variance estimator. However, when referring to a high-order RBM, since there are interactions among its visible layers, the gradient approximation via CD learning usually becomes far from the log-likelihood gradient and even may cause CD learning to fall into an infinite loop with high reconstruction error. In this paper, a new algorithm named Cyclic Contrastive Divergence (CCD) is introduced for learning high-order RBMs. Unlike the standard CD algorithm, CCD updates the parameters according to each visible layer in turn, by borrowing the idea of Cyclic Block Coordinate Descent method. To evaluate the performance of the proposed CCD algorithm, regarding to high-order RBMs learning, both algorithms CCD and standard CD are theoretically analyzed, including convergence, estimate upper bound and both biases comparison, from which the superiority of CCD learning is revealed. Experiments on MNIST dataset for the handwritten digit classification task are performed. The experimental results show that CCD is more applicable and consistently outperforms the standard CD in both convergent speed and performance. |
Year | DOI | Venue |
|---|---|---|
2014 | 10.1109/ICMLA.2014.18 | ICMLA |
Keywords | Field | DocType |
restricted boltzmann machine,statistical distributions,boltzmann machines,cyclic contrastive divergence learning algorithm,high-order rbms,convergence,approximation theory,upper bound,learning (artificial intelligence),cd learning,pattern classification,cyclic block coordinate descent method,probabilistic graphical model,cyclic contrastive divergence learning,variance estimator,ccd learning,cd algorithm,feature extraction,data point,gradient methods,gradient approximation,high-order rbm,markov chain,handwritten character recognition,reconstruction error,mnist dataset,log-likelihood gradient,markov processes,training data,handwritten digit classification task,topology,approximation algorithms,hidden markov models | MNIST database,Computer science,Artificial intelligence,Coordinate descent,Approximation algorithm,Restricted Boltzmann machine,Pattern recognition,Markov chain,Algorithm,Graphical model,Hidden Markov model,Machine learning,Estimator | Conference |
Citations | PageRank | References |
0 | 0.34 | 15 |
Authors | ||
4 | ||
Name | Order | Citations | PageRank |
|---|---|---|---|
| Dingsheng Luo | 1 | 46 | 11.61 |
| Yi Wang | 2 | 3 | 2.07 |
| Xiaoqiang Han | 3 | 1 | 0.68 |
| Xihong Wu | 4 | 279 | 53.02 |