Name
Abstract | ||
|---|---|---|
AbstractHighlights •Stage-wise optimization and momentum have been widely employed to accelerate SGD.•Negative momentum provides acceleration and stabilization on stochastic first-order methods.•Negative momentum extends Nesterovs momentum to the stage-wise optimization.•Gradient correction avoids the oscillations and make stochastic gradient more effective and tolerant. AbstractMulti-stage optimization which invokes a stochastic algorithm restarting with the returned solution of previous stage, has been widely employed in stochastic optimization. Momentum acceleration technique is famously known for building gradient-based algorithms with fast convergence in large-scale optimization. In order to take the advantage of this acceleration in multi-stage stochastic optimization, we develop a multi-stage stochastic gradient descent with momentum acceleration method, named MAGNET, for first-order stochastic convex optimization. The main ingredient is the employment of a negative momentum, which extends the Nesterov’s momentum to the multi-stage optimization. It can be incorporated in a stochastic gradient-based algorithm in multi-stage mechanism and provide acceleration. The proposed algorithm obtains an accelerated rate of convergence, and is adaptive and free from hyper-parameter tuning. The experimental results demonstrate that our algorithm is competitive with some state-of-the-art methods for solving several typical optimization problems in machine learning. |
Year | DOI | Venue |
|---|---|---|
2021 | 10.1016/j.sigpro.2021.108201 | Periodicals |
Keywords | DocType | Volume |
Multi-stage, Momentum acceleration, Stochastic gradient descent, Convex optimization | Journal | 188 |
Issue | ISSN | Citations |
C | 0165-1684 | 0 |
PageRank | References | Authors |
0.34 | 0 | 4 |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Zhijian Luo | 1 | 0 | 0.34 |
| Siyu Chen | 2 | 29 | 5.96 |
| Yuntao Qian | 3 | 597 | 54.17 |
| Yueen Hou | 4 | 0 | 0.34 |