Computer Science > Data Structures and Algorithms
[Submitted on 6 Jul 2014 (v1), revised 6 Nov 2014 (this version, v3), latest version 7 Nov 2016 (v5)]
Title:Linear Coupling of Gradient and Mirror Descent: A Novel, Simple Interpretation of Nesterov's Accelerated Method
View PDFAbstract:First-order methods play a central role in large-scale convex optimization. Even though many variations exist, each suited to a particular problem form, almost all such methods fundamentally rely on two types of algorithmic steps and two corresponding types of analysis: gradient-descent steps, which yield primal progress, and mirror-descent steps, which yield dual progress. In this paper, we observe that the performances of these two types of step are complementary, so that faster algorithms can be designed by coupling the two steps and combining their analyses.
In particular, we show how to obtain a conceptually simple interpretation of Nesterov's accelerated gradient method, a cornerstone algorithm in convex optimization. Nesterov's method is the optimal first-order method for the class of smooth convex optimization problems. However, to the best of our knowledge, the proof of the fast convergence of Nesterov's method has not found a clear interpretation and is still regarded by many as crucially relying on an "algebraic trick". We apply our novel insights to express Nesterov's algorithm as a natural coupling of gradient descent and mirror descent and to write its proof of convergence as a simple combination of the convergence analyses of the two underlying steps.
We believe that the complementary view of gradient descent and mirror descent proposed in this paper will prove very useful in the design of first-order methods as it allows us to design fast algorithms in a conceptually easier way. For instance, our view greatly facilitates the adaptation of non-trivial variants of Nesterov's method to specific scenarios, such as packing and covering problems [AO14, AO15].
Submission history
From: Zeyuan Allen-Zhu [view email][v1] Sun, 6 Jul 2014 20:11:48 UTC (540 KB)
[v2] Sat, 9 Aug 2014 01:48:01 UTC (469 KB)
[v3] Thu, 6 Nov 2014 06:59:10 UTC (467 KB)
[v4] Fri, 2 Jan 2015 17:41:24 UTC (466 KB)
[v5] Mon, 7 Nov 2016 19:30:37 UTC (439 KB)
Current browse context:
cs.DS
References & Citations
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
Connected Papers (What is Connected Papers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.