Paper Info
Title
Reducing The Variance Of Variational Estimates Of Mutual Information By Limiting The Critic'S Hypothesis Space To Rkhs
Abstract
Mutual information (MI) is an information-theoretic measure of dependency between two random variables. Several methods to estimate MI from samples of two random variables with unknown underlying probability distributions have been proposed in the literature. Recent methods realize parametric probability distributions or critic as a neural network to approximate unknown density ratios. These approximated density ratios are used to estimate different variational lower bounds of MI. While, these estimation methods are reliable when the true MI is low, they tend to produce high variance estimates when the true MI is high. We argue that the high variance characteristics is due to the uncontrolled complexity of the critic's hypothesis space. In support of this argument, we use the data-driven Rademacher complexity of the hypothesis space associated with the critic's architecture to analyse generalization error bound of variational lower bound estimates of MI. In the proposed work, we show that it is possible to negate the high variance characteristics of these estimators by constraining the critic's hypothesis space to Reproducing Hilbert Kernel Space (RKHS), which corresponds to a kernel learned using Automated Spectral Kernel Learning (ASKL). By analysing the generalization error bounds, we augment the overall optimisation objective with effective regularisation term. We empirically demonstrate the efficacy of this regularization in enforcing proper bias variance tradeoff on four different variational lower bounds of MI, namely NWJ, MINE, JS and SMILE.
Year
DOI
Venue
2020
10.1109/ICPR48806.2021.9413061
2020 25TH INTERNATIONAL CONFERENCE ON PATTERN RECOGNITION (ICPR)
DocType
ISSN
Citations 
Conference
1051-4651
0
PageRank 
References 
Authors
0.34
0
3
Name
Order
Citations
PageRank
P Aditya Sreekar110.70
Ujjwal Tiwari211.04
Anoop M. Namboodiri325526.36