Computer Science > Machine Learning
[Submitted on 23 May 2018 (v1), last revised 22 Jul 2020 (this version, v9)]
Title:Learning towards Minimum Hyperspherical Energy
View PDFAbstract:Neural networks are a powerful class of nonlinear functions that can be trained end-to-end on various applications. While the over-parametrization nature in many neural networks renders the ability to fit complex functions and the strong representation power to handle challenging tasks, it also leads to highly correlated neurons that can hurt the generalization ability and incur unnecessary computation cost. As a result, how to regularize the network to avoid undesired representation redundancy becomes an important issue. To this end, we draw inspiration from a well-known problem in physics -- Thomson problem, where one seeks to find a state that distributes N electrons on a unit sphere as evenly as possible with minimum potential energy. In light of this intuition, we reduce the redundancy regularization problem to generic energy minimization, and propose a minimum hyperspherical energy (MHE) objective as generic regularization for neural networks. We also propose a few novel variants of MHE, and provide some insights from a theoretical point of view. Finally, we apply neural networks with MHE regularization to several challenging tasks. Extensive experiments demonstrate the effectiveness of our intuition, by showing the superior performance with MHE regularization.
Submission history
From: Weiyang Liu [view email][v1] Wed, 23 May 2018 17:34:47 UTC (8,356 KB)
[v2] Tue, 5 Jun 2018 21:50:09 UTC (8,356 KB)
[v3] Wed, 13 Jun 2018 22:44:57 UTC (8,356 KB)
[v4] Sat, 16 Jun 2018 07:47:21 UTC (8,365 KB)
[v5] Sat, 27 Oct 2018 07:17:57 UTC (9,297 KB)
[v6] Sat, 1 Dec 2018 09:28:53 UTC (9,297 KB)
[v7] Wed, 9 Jan 2019 09:16:13 UTC (9,285 KB)
[v8] Tue, 5 Mar 2019 03:07:32 UTC (9,288 KB)
[v9] Wed, 22 Jul 2020 15:23:29 UTC (9,287 KB)
Current browse context:
cs.LG
References & Citations
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Connected Papers (What is Connected Papers?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
alphaXiv (What is alphaXiv?)
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Hugging Face (What is Huggingface?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
CORE Recommender (What is CORE?)
IArxiv Recommender
(What is IArxiv?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.