Computer Science > Machine Learning
[Submitted on 8 Oct 2012 (this version), latest version 7 Dec 2015 (v3)]
Title:Online Belief Propagation for Topic Modeling
View PDFAbstract:Not only can online topic modeling algorithms extract topics from big data streams with constant memory requirements, but also can detect topic shifts as the data stream flows. Fast convergence speed is a desired property for batch learning topic models such as latent Dirichlet allocation (LDA), which can further facilitate developing fast online topic modeling algorithms for big data streams. In this paper, we present a novel and easy-to-implement fast belief propagation (FBP) algorithm to accelerate the convergence speed for batch learning LDA when the number of topics is large. FBP uses a dynamic scheduling scheme for asynchronous message passing, which passes only the most important subset of topic messages at each iteration for fast speed. From FBP, we derive an online belief propagation (OBP) algorithm that infers the topic distribution from the previously unseen documents incrementally by the online gradient descent. We show that OBP can converge to the local optimum of the LDA objective function within the online stochastic optimization framework. Extensive empirical studies demonstrate that OBP significantly reduces the learning time and achieves a much lower predictive perplexity when compared with that of several state-of-the-art online algorithms for LDA, including online variational Bayes (OVB) and online Gibbs sampling (OGS) algorithms.
Submission history
From: Jia Zeng [view email][v1] Mon, 8 Oct 2012 08:17:18 UTC (5,361 KB)
[v2] Thu, 13 Jun 2013 04:17:05 UTC (5,107 KB)
[v3] Mon, 7 Dec 2015 13:49:04 UTC (3,721 KB)
References & Citations
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Connected Papers (What is Connected Papers?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
alphaXiv (What is alphaXiv?)
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Hugging Face (What is Huggingface?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
CORE Recommender (What is CORE?)
IArxiv Recommender
(What is IArxiv?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.