Title
Minimum Entropy Rate Simplification of Stochastic Processes
Abstract
We propose minimum entropy rate simplification (MERS), an information-theoretic, parameterization-independent framework for simplifying generative models of stochastic processes. Applications include improving model quality for sampling tasks by concentrating the probability mass on the most characteristic and accurately described behaviors while de-emphasizing the tails, and obtaining clean models from corrupted data (nonparametric denoising). This is the opposite of the smoothing step commonly applied to classification models. Drawing on rate-distortion theory, MERS seeks the minimum entropy-rate process under a constraint on the dissimilarity between the original and simplified processes. We particularly investigate the Kullback-Leibler divergence rate as a dissimilarity measure, where, compatible with our assumption that the starting model is disturbed or inaccurate, the simplification rather than the starting model is used for the reference distribution of the divergence. This leads to analytic solutions for stationary and ergodic Gaussian processes and Markov chains. The same formulas are also valid for maximum-entropy smoothing under the same divergence constraint. In experiments, MERS successfully simplifies and denoises models from audio, text, speech, and meteorology.
Year
DOI
Venue
2016
10.1109/TPAMI.2016.2533382
IEEE Trans. Pattern Anal. Mach. Intell.
Keywords
Field
DocType
Markov processes,Stochastic processes,Gaussian processes,Electronic mail,Distortion,Density functional theory,Signal processing
Applied mathematics,Markov process,Computer science,Theoretical computer science,Gaussian process,Artificial intelligence,Information theory,Probability mass function,Pattern recognition,Markov chain,Stochastic process,Smoothing,Statistical model
Journal
Volume
Issue
ISSN
38
12
0162-8828
Citations 
PageRank 
References 
0
0.34
16
Authors
2
Name
Order
Citations
PageRank
Gustav Eje Henter13711.40
W. Bastiaan Kleijn21110106.92