Quantitative Biology > Neurons and Cognition
[Submitted on 28 Feb 2018 (this version), latest version 24 Jun 2020 (v4)]
Title:Optimal encoding in stochastic latent-variable Models
View PDFAbstract:We examine the problem of optimal sparse encoding of sensory stimuli by latent variables in stochastic models. Analyzing restricted Boltzmann machines with a communications theory approach, we search for the minimal model size that correctly conveys the correlations in stimulus patterns in an information-theoretic sense. We show that the Fisher information Matrix (FIM) reveals the optimal model size. In larger models the FIM reveals that irrelevant parameters are associated with individual latent variables, displaying a surprising amount of order. For well-fit models, we observe the emergence of statistical criticality as diverging generalized susceptibility of the model. In this case, an encoding strategy is adopted where highly informative, but rare stimuli selectively suppress variability in the encoding units. The information content of the encoded stimuli acts as an unobserved variable leading to criticality. Together, these results can explain the stimulus-dependent variability suppression observed in sensory systems, and suggest a simple, correlation-based measure to reduce the size of artificial neural networks.
Submission history
From: M Rule [view email][v1] Wed, 28 Feb 2018 11:17:13 UTC (1,205 KB)
[v2] Mon, 5 Mar 2018 14:28:59 UTC (1,366 KB)
[v3] Mon, 25 May 2020 10:33:20 UTC (2,290 KB)
[v4] Wed, 24 Jun 2020 06:16:13 UTC (5,818 KB)
References & Citations
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
Connected Papers (What is Connected Papers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.