Computer Science > Data Structures and Algorithms
[Submitted on 6 Jul 2014 (this version), latest version 7 Nov 2016 (v5)]
Title:A Novel, Simple Interpretation of Nesterov's Accelerated Method as a Combination of Gradient and Mirror Descent
View PDFAbstract:First-order methods play a central role in large-scale convex optimization. Despite their various forms of descriptions and many applications, such methods mostly and fundamentally rely on two basic types of analyses: gradient-descent analysis, which yields primal progress, and mirror-descent analysis, which yields dual progress. In this paper, we observe that the performances of these two analyses are complementary, so that faster algorithms can be designed by coupling the two analyses, and their corresponding descent steps.
In particular, we show in this paper how to obtain a conceptually simple reinterpretation of Nesterov's accelerated gradient method [Nes83, Nes04, Nes05]. Nesterov's method is the optimal first-order method for the class of smooth convex optimization problems. However, the proof of the fast convergence of Nesterov's method has no clear interpretation and is regarded by some as relying on an `algebraic trick'. We apply our novel insights to express Nesterov's algorithm as a coupling of gradient descent and mirror descent, and as a result, the convergence proof can be understood as some natural combination of the two underlying convergence analyses.
We believe that this complementary view of the two types of analysis may not only facilitate the study of Nesterov's method in a white-box manner so as to apply it to problems outside its original scope, but also let us design better first-order methods in a conceptually easier way.
Submission history
From: Zeyuan Allen-Zhu [view email][v1] Sun, 6 Jul 2014 20:11:48 UTC (540 KB)
[v2] Sat, 9 Aug 2014 01:48:01 UTC (469 KB)
[v3] Thu, 6 Nov 2014 06:59:10 UTC (467 KB)
[v4] Fri, 2 Jan 2015 17:41:24 UTC (466 KB)
[v5] Mon, 7 Nov 2016 19:30:37 UTC (439 KB)
Current browse context:
cs.DS
References & Citations
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
Connected Papers (What is Connected Papers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.