Title:RSG: Beating SG without Smoothness and/or Strong Convexity

Abstract:In this paper, we propose novel deterministic and stochastic {\bf R}estarted {\bf S}ub{\bf G}radient (RSG) methods that can find an $\epsilon$-optimal solution for a broad class of non-smooth and/or non-strongly convex optimization problems faster than the vanilla deterministic or stochastic subgradient method (SG). We show that for non-smooth and non-strongly convex optimization, RSG can reduce the dependence of SG's iteration complexity on the distance to the optimal set of the initial solution to that of points on the $\epsilon$-level set. For a special family of non-smooth and non-strongly convex optimization problems whose epigraph is a polyhedron, we further show that RSG could converge linearly. In addition, RSG has an $O(\frac{1}{\epsilon}\log(\frac{1}{\epsilon}))$ iteration complexity for problems with a much weaker notion of strong convexity, namely locally semi-strongly convexity. For a family of non-smooth optimization problems that admit a local Kurdyka-Łojasiewicz property with a power constant of $\beta\in(0,1)$, RSG has an $O(\frac{1}{\epsilon^{2\beta}}\log(\frac{1}{\epsilon}))$ iteration complexity, which is better than that of SG for such optimization problems whose iteration complexity is $O(\frac{1}{\epsilon^2})$. The novelty of our analysis lies at exploiting the lower bound of the subgradient of the objective function at the $\epsilon$-level set. It is this novelty that allows us to explore the local properties of functions (e.g., local semi-strong convexity, local Kurdyka-Łojasiewicz property, more generally local error bounds) to develop improved convergence of RSG.