Name
Title | ||
|---|---|---|
Exploring one pass learning for deep neural network training with averaged stochastic gradient descent |
Abstract | ||
|---|---|---|
Deep neural network acoustic models have shown large improvement in performance over Gaussian mixture models (G-MMs) in recent studies. Typically, deep neural networks are trained based on the cross-entropy criterion using stochastic gradient descent (SGD). However, plain SGD requires scanning the whole training set many passes before reaching the asymptotic region, making it difficult to scale to large dataset. It has been established that the second order SGD can potentially reach its asymptotic region in one pass through the training dataset. However, since it involves expensive computing for the inverse of Hessian matrix in the loss function, its application is limited. Averaged stochastic gradient descent (ASGD) is proved simple and effective for one pass online learning. This paper investigates the ASGD algorithm for deep neural network training. We tested ASGD on the Mandarin Chinese record speech recognition task using deep neural networks. Experimental results show that the performance of one pass ASGD is very close to that of multiple passes SGD. |
Year | DOI | Venue |
|---|---|---|
2014 | 10.1109/ICASSP.2014.6854928 | ICASSP |
Keywords | Field | DocType |
one pass learning,hessian matrices,deep neural network,asgd,speech recognition,acoustic models,mandarin chinese record speech recognition task,asymptotic region,expensive computing,deep neural network training,mixture models,averaged stochastic gradient descent,g-mm,telecommunication computing,training dataset,acoustic signal processing,gaussian processes,inverse hessian matrix,gaussian mixture models,entropy,cross-entropy criterion,neural nets,neural networks,optimization,schedules,stochastic processes,acoustics | Online learning,Training set,Inverse,Stochastic gradient descent,Pattern recognition,Computer science,Hessian matrix,Artificial intelligence,Artificial neural network,Mixture model,Deep neural networks | Conference |
ISSN | Citations | PageRank |
1520-6149 | 1 | 0.35 |
References | Authors | |
5 | 3 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Zhao You | 1 | 67 | 9.39 |
| Xiaorui Wang | 2 | 19 | 6.13 |
| Bo Xu | 3 | 241 | 36.59 |