Mathematics > Optimization and Control
[Submitted on 9 Sep 2020 (this version), latest version 27 Aug 2022 (v3)]
Title:Optimal Accelerated Variance Reduced EXTRA and DIGing for Strongly Convex and Smooth Decentralized Optimization
View PDFAbstract:We study stochastic decentralized optimization for the problem of training machine learning models with large-scale distributed data. We extend the famous EXTRA and DIGing methods with accelerated variance reduction (VR), and propose two methods, which require the time of $O((\sqrt{n\kappa_s}+n)\log\frac{1}{\epsilon})$ stochastic gradient evaluations and $O(\sqrt{\kappa_b\kappa_c}\log\frac{1}{\epsilon})$ communication rounds to reach precision $\epsilon$, where $\kappa_s$ and $\kappa_b$ are the stochastic condition number and batch condition number for strongly convex and smooth problems, $\kappa_c$ is the condition number of the communication network, and $n$ is the sample size on each distributed node. Our stochastic gradient computation complexity is the same as the single-machine accelerated variance reduction methods, such as Katyusha, and our communication complexity is the same as the accelerated full batch decentralized methods, such as MSDA, and they are both optimal. We also propose the non-accelerated VR based EXTRA and DIGing, and provide explicit complexities, for example, the $O((\kappa_s+n)\log\frac{1}{\epsilon})$ stochastic gradient computation complexity and the $O((\kappa_b+\kappa_c)\log\frac{1}{\epsilon})$ communication complexity for the VR based EXTRA. The two complexities are also the same as the ones of single-machine VR methods, such as SAG, SAGA, and SVRG, and the non-accelerated full batch decentralized methods, such as EXTRA, respectively.
Submission history
From: Huan Li [view email][v1] Wed, 9 Sep 2020 15:48:44 UTC (44 KB)
[v2] Sat, 10 Oct 2020 13:09:53 UTC (378 KB)
[v3] Sat, 27 Aug 2022 12:55:53 UTC (355 KB)
Current browse context:
math.OC
References & Citations
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
Connected Papers (What is Connected Papers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.