Statistics > Methodology
[Submitted on 9 May 2015 (v1), revised 12 May 2015 (this version, v2), latest version 30 Aug 2021 (v4)]
Title:Measuring dependence powerfully and equitably
View PDFAbstract:For high-dimensional datasets, it is common to evaluate a measure of dependence on every variable pair and retain the highest-scoring pairs for follow-up. If the statistic used systematically assigns higher scores to some relationship types over others, important relationships may be overlooked. This difficulty is avoided if the statistic is equitable [Reshef et al. 2015a], i.e., if, for some measure of noise, it assigns similar scores to equally noisy relationships regardless of relationship type.
In this paper, we introduce and characterize a population measure of dependence called MIC*. We show three ways that MIC* can be viewed: as the population value of MIC, a highly equitable statistic from [Reshef et al. 2011], as a canonical "smoothing" of mutual information, and as the supremum of an infinite sequence defined in terms of optimal one-dimensional partitions of the marginals of the joint distribution. Based on this theory, we introduce an efficient algorithm for computing MIC* from the density of a pair of random variables, and we define a new consistent estimator MICe for MIC* that is efficiently computable. (In contrast, there is no known polynomial-time algorithm for computing MIC.) We show through simulations that MICe has better bias-variance properties than MIC, and that it has high equitability with respect to R^2 on a set of functional relationships. While MICe is designed for equitability rather than independence testing, we introduce a related statistic, TICe, that is a trivial side-product of the computation of MICe. We prove the consistency of independence testing based on TICe and show in simulations that this approach achieves excellent power.
This paper is accompanied by a companion paper [Reshef et al. 2015b] focused on in-depth empirical evaluation of several leading measures of dependence that finds that the performance of MICe and TICe is state-of-the-art.
Submission history
From: Yakir Reshef [view email][v1] Sat, 9 May 2015 00:31:54 UTC (2,233 KB)
[v2] Tue, 12 May 2015 19:51:46 UTC (2,233 KB)
[v3] Wed, 6 Jul 2016 18:42:36 UTC (3,745 KB)
[v4] Mon, 30 Aug 2021 20:58:59 UTC (3,745 KB)
Current browse context:
stat.ME
References & Citations
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Connected Papers (What is Connected Papers?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
alphaXiv (What is alphaXiv?)
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Hugging Face (What is Huggingface?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.