Name
Title | ||
|---|---|---|
A Quasi-Newton Approach to Nonsmooth Convex Optimization Problems in Machine Learning |
Abstract | ||
|---|---|---|
We extend the well-known BFGS quasi-Newton method and its memory-limited variant LBFGS to the optimization of nonsmooth convex objectives. This is done in a rigorous fashion by generalizing three components of BFGS to subdifferentials: the local quadratic model, the identification of a descent direction, and the Wolfe line search conditions. We prove that under some technical conditions, the resulting subBFGS algorithm is globally convergent in objective function value. We apply its memory-limited variant (subLBFGS) to L2-regularized risk minimization with the binary hinge loss. To extend our algorithm to the multiclass and multilabel settings, we develop a new, efficient, exact line search algorithm. We prove its worst-case time complexity bounds, and show that our line search can also be used to extend a recently developed bundle method to the multiclass and multilabel settings. We also apply the direction-finding component of our algorithm to L1-regularized risk minimization with logistic loss. In all these contexts our methods perform comparable to or better than specialized state-of-the-art solvers on a number of publicly available data sets. An open source implementation of our algorithms is freely available. |
Year | DOI | Venue |
|---|---|---|
2010 | 10.1145/1756006.1756045 | Journal of Machine Learning Research |
Keywords | Field | DocType |
bundle method,multilabel setting,nonsmooth convex optimization problems,machine learning,available data set,l1-regularized risk minimization,subbfgs algorithm,logistic loss,quasi-newton approach,l2-regularized risk minimization,exact line search algorithm,wolfe line search condition,line search,quasi newton method,time complexity,convex optimization,objective function | Mathematical optimization,Hinge loss,Generalization,Descent direction,Line search,Minification,Artificial intelligence,Time complexity,Broyden–Fletcher–Goldfarb–Shanno algorithm,Convex optimization,Mathematics,Machine learning | Journal |
Volume | ISSN | Citations |
11, | Journal of Machine Learning Research 11(Mar):1145-1200, 2010 | 32 |
PageRank | References | Authors |
1.90 | 21 | 4 |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Jin Yu | 1 | 86 | 3.68 |
| S. V. N. Vishwanathan | 2 | 1991 | 131.90 |
| Simon Günter | 3 | 588 | 34.93 |
| Nicol N. Schraudolph | 4 | 1185 | 164.26 |