Statistics > Computation
[Submitted on 29 Oct 2018 (v1), revised 9 Dec 2018 (this version, v2), latest version 3 Mar 2022 (v6)]
Title:Prior-preconditioned conjugate gradient method for accelerated Gibbs sampling in "large n & large p" sparse Bayesian regression
View PDFAbstract:In a modern observational study based on healthcare databases, the number of observations typically ranges in the order of 10^5 ~ 10^6 and that of the predictors in the order of 10^4 ~ 10^5. Despite the large sample size, data rarely provide sufficient information to reliably estimate such a large number of parameters. Sparse regression provides a potential solution. Bayesian approaches based on shrinkage priors possess many desirable theoretical properties and, under linear and logistic models, yield posterior distributions amenable to Gibbs sampling. A major computational bottleneck arises in the "large n & large p" setting, however, from the need to sample from a high-dimensional Gaussian distribution at each iteration; despite the availability of a closed-form expression for the precision matrix $\Phi$, computing and factorizing such a large matrix is computationally expensive nonetheless. In this article, we present a novel algorithm to speed up this bottleneck based on the following observation: we can cheaply generate a random vector $b$ such that the solution to the linear system $\Phi \beta = b$ has the desired Gaussian distribution. We can then solve the linear system by the conjugate gradient (CG) algorithm through the matrix-vector multiplications by $\Phi$, without ever explicitly inverting $\Phi$. As practical performance of CG depends critically on appropriate preconditioning of the linear system, we develop a theory of prior-preconditioning to turn CG into a highly effective algorithm for sparse Bayesian regression. We apply our algorithm to a clinically relevant large-scale observational study with n = 72,489 and p = 22,175, designed to assess the relative risk of intracranial hemorrhage from two alternative blood anti-coagulants. Our algorithm demonstrates an order of magnitude speed-up in the posterior computation.
Submission history
From: Akihiko Nishimura [view email][v1] Mon, 29 Oct 2018 22:21:56 UTC (484 KB)
[v2] Sun, 9 Dec 2018 15:24:01 UTC (1,229 KB)
[v3] Mon, 4 Mar 2019 16:44:40 UTC (1,231 KB)
[v4] Fri, 17 Jan 2020 17:35:58 UTC (1,224 KB)
[v5] Tue, 20 Jul 2021 18:01:22 UTC (940 KB)
[v6] Thu, 3 Mar 2022 19:05:48 UTC (1,789 KB)
Current browse context:
stat.CO
References & Citations
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Connected Papers (What is Connected Papers?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
alphaXiv (What is alphaXiv?)
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Hugging Face (What is Huggingface?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.