Name
Title | ||
|---|---|---|
AN OPTIMAL STOCHASTIC COMPOSITIONAL OPTIMIZATION METHOD WITH APPLICATIONS TO META LEARNING |
Abstract | ||
|---|---|---|
Stochastic compositional optimization generalizes classic (non-compositional) stochastic optimization to the minimization of compositions of functions. Each composition may introduce an additional expectation. The series of expectations may be nested. Stochastic compositional optimization is gaining popularity in applications such as meta learning. This paper presents a new Stochastically Corrected Stochastic Compositional gradient method (SCSC). SCSC runs in a single-time scale with a single loop, uses a fixed batch size, and guarantees to converge at the same rate as the stochastic gradient descent (SGD) method for non-compositional stochastic optimization. It is easy to apply SGD-improvement techniques to accelerate SCSC. This helps SCSC achieve state-of-the-art performance for stochastic compositional optimization. In particular, we apply Adam to SCSC, and the exhibited rate of convergence matches that of the original Adam on non-compositional optimization. We test SCSC using the model-agnostic meta-learning tasks. |
Year | DOI | Venue |
|---|---|---|
2021 | 10.1109/ICASSP39728.2021.9414369 | 2021 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP 2021) |
Keywords | DocType | Citations |
Stochastic optimization, compositional optimization, stochastic gradient descent, model-agnostic meta-learning | Conference | 0 |
PageRank | References | Authors |
0.34 | 0 | 3 |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Yuejiao Sun | 1 | 2 | 2.09 |
| Tianyi Chen | 2 | 43 | 7.52 |
| Yin, Wotao | 3 | 13 | 1.36 |