Paper Info
Title
Information estimators for weighted observations.
Abstract
The Shannon information content is a valuable numerical characteristic of probability distributions. The problem of estimating the information content from an observed dataset is very important in the fields of statistics, information theory, and machine learning. The contribution of the present paper is in proposing information estimators, and showing some of their applications. When the given data are associated with weights, each datum contributes differently to the empirical average of statistics. The proposed estimators can deal with this kind of weighted data. Similar to other conventional methods, the proposed information estimator contains a parameter to be tuned, and is computationally expensive. To overcome these problems, the proposed estimator is further modified so that it is more computationally efficient and has no tuning parameter. The proposed methods are also extended so as to estimate the cross-entropy, entropy, and Kullback-Leibler divergence. Simple numerical experiments show that the information estimators work properly. Then, the estimators are applied to two specific problems, distribution-preserving data compression, and weight optimization for ensemble regression.
Year
DOI
Venue
2013
10.1016/j.neunet.2013.06.005
Neural Networks
Keywords
Field
DocType
information estimator,entropy estimation,weighted data,distribution-preserving data compression,information content,weighted observation,information estimation,information theory,proposed information estimator,simple numerical experiment,proposed estimator,shannon information content
Information theory,Entropy estimation,Extremum estimator,Regression,Probability distribution,Artificial intelligence,Data compression,Entropy (information theory),Machine learning,Mathematics,Estimator
Journal
Volume
Issue
ISSN
46
1
1879-2782
Citations 
PageRank 
References 
5
0.52
20
Authors
2
Name
Order
Citations
PageRank
Hideitsu Hino19925.73
Noboru Murata2855170.36