Paper Info
Title
Ordalia: Deep Learning Hyperparameter Search via Generalization Error Bounds Extrapolation
Abstract
We introduce Ordalia, a novel approach for speeding up deep learning hyperparameter optimization search through early-pruning of less promising configurations. Our method leverages empirical and theoretical results characterizing the shape of the generalization error curve for increasing training data size and number of epochs. We show that with relatively small computational resources one can estimate the dominant parameters of neural networks' learning curves to obtain consistently good evaluations of their learning process to reliably early-eliminate non-promising configurations. By iterating this process with increasing training resources Ordalia rapidly converges to a small candidate set that includes many of the most promising configurations. We compare the performance of Ordalia with Hyperband, the state-of-the-art model-free hyperparameter optimization algorithm, and show that Ordalia consistently outperforms it on a variety of deep learning tasks. Ordalia conservative use of computational resources and ability to evaluate neural networks learning progress leads to a much better exploration and coverage of the search space, which ultimately produces superior neural network configurations.
Year
DOI
Venue
2019
10.1109/BigData47090.2019.9006144
2019 IEEE International Conference on Big Data (Big Data)
Keywords
Field
DocType
Deep Learning,Hyperparameters Optimization,Multi-armed Bandits,Automated Machine Learning
Hyperparameter optimization,Learning progress,Hyperparameter,Computer science,Extrapolation,Artificial intelligence,Generalization error,Deep learning,Artificial neural network,Learning curve,Machine learning
Conference
ISSN
ISBN
Citations 
2639-1589
978-1-7281-0859-9
0
PageRank 
References 
Authors
0.34
0
2
Name
Order
Citations
PageRank
Benedetto Buratti152.10
Eli Upfal24310743.13