Name
Abstract | ||
|---|---|---|
As deep learning has been widespread in a wide range of applications, its training speed and convergence have become crucial. Among different hyperparameters existed in the gradient descent algorithm, the learning rate has an essential role in the learning procedure. This paper presents a new statistical algorithm for adapting the learning rate during the training process. The proposed T-LRA (trend-based learning rate annealing) algorithm is calculated based on the statistical trends seen in the previous training iterations. The proposed algorithm is computationally very cheap and applicable to online training for very deep networks and large datasets. This efficient, simple, and well-principled algorithm not only improves the deep learning results, but also speeds up the training convergence. Experimental results on a multimedia dataset and deep learning networks demonstrate the effectiveness and efficiency of the proposed algorithm. |
Year | DOI | Venue |
|---|---|---|
2017 | 10.1109/BigMM.2017.36 | 2017 IEEE Third International Conference on Multimedia Big Data (BigMM) |
Keywords | Field | DocType |
Learning rate annealing,Deep learning,Convolutional Neural Networks (CNNs),Statistical trend analysis | Online machine learning,Competitive learning,Stability (learning theory),Computer science,Wake-sleep algorithm,Supervised learning,Unsupervised learning,Artificial intelligence,Artificial neural network,Population-based incremental learning,Machine learning | Conference |
ISBN | Citations | PageRank |
978-1-5090-6550-9 | 1 | 0.35 |
References | Authors | |
22 | 2 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Samira Pouyanfar | 1 | 141 | 13.06 |
| Shu-Ching Chen | 2 | 1978 | 182.74 |