Title:Restarted SGD: Beating SGD without Smoothness and/or Strong Convexity

Abstract:In this paper, we show that a restarted stochastic (sub)gradient {\bf d}escent (RSGD) can find an $\epsilon$-optimal solution for a broad class of non-smooth and/or non-strongly convex optimization problems with a lower iteration complexity than vanilla stochastic subgradient descent (SGD). Of foremost interest, we show that for non-smooth and non-strongly convex optimization, RSGD reduces the dependence of SGD's iteration complexity on the distance of initial solution to the optimal set to the dependence on the maximum distance of points in the $\epsilon$-level set to the optimal set. This result of RSGD for non-smooth and non-strongly convex problems complements the previous work~\cite{arXiv:1510.01444} that establishes a linear convergence rate for RSGD for a special family of non-smooth and non-strongly convex optimization problems whose epigraph is a polyhedron. Besides that, we also obtain some interesting results of RSGD for locally semi-strongly convex problems and smooth optimization problems that have a Łojasiewicz property. In particular, (i) for locally semi-strongly convex functions, RSGD has an $O(\frac{1}{\epsilon}\log(\frac{1}{\epsilon}))$ iteration complexity, which in the strongly convex scenario is similar to if not better than that of SGD; (ii) for smooth optimization problems that have a Łojasiewicz property with a power constant of $\beta\in(0,1)$, RSGD~\footnote{In this case, RSGD refers to restarted stochastic gradient descent.} has an $O(\frac{1}{\epsilon^{2\beta}}\log(\frac{1}{\epsilon}))$ iteration complexity, which is better that that of SGD for such smooth optimization problems whose iteration complexity is $O(\frac{1}{\epsilon^2})$. The novelty of our analysis lies at exploiting the lower bound of the subgradient or gradient of the objection function at the $\epsilon$-level set.