Name
Abstract | ||
|---|---|---|
Information bottleneck (IB) is a technique for extracting information in one random variable X that is relevant for predicting another random variable Y. IB works by encoding X in a compressed "bottleneck" random variable M from which Y can be accurately decoded. However, finding the optimal bottleneck variable involves a difficult optimization problem, which until recently has been considered for only two limited cases: discrete X and Y with small state spaces, and continuous X and Y with a Gaussian joint distribution (in which case optimal encoding and decoding maps are linear). We propose a method for performing IB on arbitrarily-distributed discrete and/or continuous X and Y, while allowing for nonlinear encoding and decoding maps. Our approach relies on a novel non-parametric upper bound for mutual information. We describe how to implement our method using neural networks. We then show that it achieves better performance than the recently-proposed "variational IB" method on several real-world datasets. |
Year | DOI | Venue |
|---|---|---|
2019 | 10.3390/e21121181 | ENTROPY |
Keywords | DocType | Volume |
information bottleneck,mutual information,representation learning,neural networks | Journal | 21 |
Issue | Citations | PageRank |
12 | 3 | 0.40 |
References | Authors | |
0 | 3 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Artemy Kolchinsky | 1 | 71 | 8.85 |
| Brendan Tracey | 2 | 64 | 6.94 |
| David H. Wolpert | 3 | 4334 | 591.07 |