Stochastic Gradient Descent Escapes Saddle Points Efficiently
Jin, Chi ; Netrapalli, Praneeth ; Ge, Rong ; Kakade, Sham M. ; Jordan, Michael I.
arXiv, Tome 2019 (2019) no. 0, / Harvested from
This paper considers the perturbed stochastic gradient descent algorithm and shows that it finds $\epsilon$-second order stationary points ($\left\|\nabla f(x)\right\|\leq \epsilon$ and $\nabla^2 f(x) \succeq -\sqrt{\epsilon} \mathbf{I}$) in $\tilde{O}(d/\epsilon^4)$ iterations, giving the first result that has linear dependence on dimension for this setting. For the special case, where stochastic gradients are Lipschitz, the dependence on dimension reduces to polylogarithmic. In addition to giving new results, this paper also presents a simplified proof strategy that gives a shorter and more elegant proof of previously known results (Jin et al. 2017) on perturbed gradient descent algorithm.
Publié le : 2019-02-13
Classification:  Computer Science - Machine Learning,  Mathematics - Optimization and Control,  Statistics - Machine Learning
@article{1902.04811,
     author = {Jin, Chi and Netrapalli, Praneeth and Ge, Rong and Kakade, Sham M. and Jordan, Michael I.},
     title = {Stochastic Gradient Descent Escapes Saddle Points Efficiently},
     journal = {arXiv},
     volume = {2019},
     number = {0},
     year = {2019},
     language = {en},
     url = {http://dml.mathdoc.fr/item/1902.04811}
}
Jin, Chi; Netrapalli, Praneeth; Ge, Rong; Kakade, Sham M.; Jordan, Michael I. Stochastic Gradient Descent Escapes Saddle Points Efficiently. arXiv, Tome 2019 (2019) no. 0, . http://gdmltest.u-ga.fr/item/1902.04811/