Name
Title | ||
|---|---|---|
One Method to Rule Them All: Variance Reduction for Data, Parameters and Many New Methods. |
Abstract | ||
|---|---|---|
We propose a remarkably general variance-reduced method suitable for solving regularized empirical risk minimization problems with either a large number of training examples, or a large model dimension, or both. In special cases, our method reduces to several known and previously thought to be unrelated methods, such as {\tt SAGA}, {\tt LSVRG}, {\tt JacSketch}, {\tt SEGA} and {\tt ISEGA}, and their arbitrary sampling and proximal generalizations. However, we also highlight a large number of new specific algorithms with interesting properties. We provide a single theorem establishing linear convergence of the method under smoothness and quasi strong convexity assumptions. With this theorem we recover best-known and sometimes improved rates for known methods arising in special cases. As a by-product, we provide the first unified method and theory for stochastic gradient and stochastic coordinate descent type methods. |
Year | Venue | DocType |
|---|---|---|
2019 | arXiv: Optimization and Control | Journal |
Volume | Citations | PageRank |
abs/1905.11266 | 0 | 0.34 |
References | Authors | |
0 | 2 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Filip Hanzely | 1 | 5 | 4.80 |
| Peter Richtárik | 2 | 1314 | 84.53 |