Computer Science > Machine Learning
[Submitted on 24 Aug 2023 (v1), revised 28 Feb 2024 (this version, v5), latest version 28 Mar 2024 (v6)]
Title:FedSOL: Stabilized Orthogonal Learning in Federated Learning
View PDF HTML (experimental)Abstract:Federated Learning(FL) aggregates locally trained models from individual clients to construct a global model. While FL enables learning a model with data privacy, it often suffers from significant performance degradation when clients have heterogeneous data distributions. This data heterogeneity causes the model to forget the global knowledge acquired from previously sampled clients after being trained on local datasets. Although the introduction of proximal objectives in local updates helps to preserve global knowledge, it can also hinder local learning by interfering with local objectives. To address this problem, we propose a novel method, Federated Stabilized Orthogonal Learning(FedSOL), which adopts an orthogonal learning strategy to balance the two conflicting objectives. FedSOL is designed to identify gradients of local objectives that are inherently orthogonal to directions affecting the proximal objective. Specifically, FedSOL targets parameter regions where learning on the local objective is minimally influenced by proximal weight perturbations. Our experiments demonstrate that FedSOL consistently achieves state-of-the-art performance across various scenarios.
Submission history
From: Gihun Lee [view email][v1] Thu, 24 Aug 2023 03:43:02 UTC (2,883 KB)
[v2] Mon, 27 Nov 2023 03:33:37 UTC (4,745 KB)
[v3] Tue, 28 Nov 2023 03:42:45 UTC (4,745 KB)
[v4] Tue, 27 Feb 2024 01:21:16 UTC (4,745 KB)
[v5] Wed, 28 Feb 2024 06:56:20 UTC (4,918 KB)
[v6] Thu, 28 Mar 2024 08:23:02 UTC (4,921 KB)
Current browse context:
cs.LG
References & Citations
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
Connected Papers (What is Connected Papers?)
CORE Recommender (What is CORE?)
IArxiv Recommender
(What is IArxiv?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.