Name
Title | ||
|---|---|---|
Type 1 and 2 mixtures of Kullback-Leibler divergences as cost functions in dimensionality reduction based on similarity preservation |
Abstract | ||
|---|---|---|
Stochastic neighbor embedding (SNE) and its variants are methods of dimensionality reduction (DR) that involve normalized softmax similarities derived from pairwise distances. These methods try to reproduce in the low-dimensional embedding space the similarities observed in the high-dimensional data space. Their outstanding experimental results, compared to previous state-of-the-art methods, originate from their capability to foil the curse of dimensionality. Previous work has shown that this immunity stems partly from a property of shift invariance that allows appropriately normalized softmax similarities to mitigate the phenomenon of norm concentration. This paper investigates a complementary aspect, namely, the cost function that quantifies the mismatch between similarities computed in the high- and low-dimensional spaces. Stochastic neighbor embedding and its variant t-SNE rely on a single Kullback-Leibler divergence, whereas a weighted mixture of two dual KL divergences is used in neighborhood retrieval and visualization (NeRV). We propose in this paper a different mixture of KL divergences, which is a scaled version of the generalized Jensen-Shannon divergence. We show experimentally that this divergence produces embeddings that better preserve small K-ary neighborhoods, as compared to both the single KL divergence used in SNE and t-SNE and the mixture used in NeRV. These results allow us to conclude that future improvements in similarity-based DR will likely emerge from better definitions of the cost function. |
Year | DOI | Venue |
|---|---|---|
2013 | 10.1016/j.neucom.2012.12.036 | Neurocomputing |
Keywords | Field | DocType |
kl divergence,weighted mixture,single kl divergence,generalized jensen-shannon divergence,low-dimensional embedding space,stochastic neighbor embedding,different mixture,dual kl divergence,cost function,single kullback-leibler divergence,similarity preservation,dimensionality reduction,manifold learning,divergence | Pairwise comparison,Dimensionality reduction,Embedding,Divergence,Softmax function,Pattern recognition,Curse of dimensionality,Artificial intelligence,Nonlinear dimensionality reduction,Kullback–Leibler divergence,Machine learning,Mathematics | Journal |
Volume | ISSN | Citations |
112, | 0925-2312 | 26 |
PageRank | References | Authors |
1.13 | 26 | 5 |
Name | Order | Citations | PageRank |
|---|---|---|---|
| John A. Lee | 1 | 308 | 28.72 |
| Emilie Renard | 2 | 26 | 1.47 |
| Guillaume Bernard | 3 | 40 | 5.23 |
| Pierre Dupont | 4 | 380 | 29.30 |
| Michel Verleysen | 5 | 2291 | 221.75 |