Title:Restarted Stochastic subGradient Descent for Non-smooth Non-strongly Convex Optimization

Abstract:In this paper, we show that a restarted stochastic subgradient descent (RSGD) can find an $\epsilon$-optimal solution for a broad class of non-smooth non-strongly convex optimization problems with a lower iteration complexity than vanilla stochastic subgradient descent. The RSGD works by running SGD with a fixed number of iterations and restarting it using the averaged solution from the previous epoch as an initial solution and a gradually decreasing step size. Our novel analysis is based on a crucial fact that the distance of any point to the $\epsilon$-level set is bounded by a multiple of the difference between its objective value and the objective value at the $\epsilon$-level set, which is established by exploiting the lower bound of subgradients of the objective function at the $\epsilon$-level set. This work complements the previous work~\cite{arXiv:1510.01444} that establishes a linear convergence rate for RSGD for a special family of non-smooth and non-strongly convex optimization problems whose epigraph is a polyhedron.