Paper Info
Title
Exponential Family Estimation via Adversarial Dynamics Embedding.
Abstract
We present an efficient algorithm for maximum likelihood estimation (MLE) of exponential family models, with a general parametrization of the energy function that includes neural networks. We exploit the primal-dual view of the MLE with a kinetics augmented model to obtain an estimate associated with an adversarial dual sampler. To represent this sampler, we introduce a novel neural architecture, dynamics embedding, that generalizes Hamiltonian Monte-Carlo (HMC). The proposed approach inherits the flexibility of HMC while enabling tractable entropy estimation for the augmented model. By learning both a dual sampler and the primal model simultaneously, and sharing parameters between them, we obviate the requirement to design a separate sampling procedure once the model has been trained, leading to more effective learning. We show that many existing estimators, such as contrastive divergence, pseudo/composite-likelihood, score matching, minimum Stein discrepancy estimator, non-local contrastive objectives, noise-contrastive estimation, and minimum probability flow, are special cases of the proposed approach, each expressed by a different (fixed) dual sampler. An empirical investigation shows that adapting the sampler during MLE can significantly improve on state-of-the-art estimators(1).
Year
Venue
Keywords
2019
ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 32 (NIPS 2019)
maximum likelihood estimation (mle),neural networks,noise-contrastive estimation
Field
DocType
Volume
Entropy estimation,Mathematical optimization,Embedding,Parametrization,Hamiltonian (quantum mechanics),Exponential family,Algorithm,Artificial neural network,Mathematics,Adversarial system,Estimator
Journal
32
ISSN
Citations 
PageRank 
1049-5258
0
0.34
References 
Authors
0
7
Name
Order
Citations
PageRank
Bo Dai123034.71
Zhen Liu2405.01
Hanjun Dai332325.71
Niao He421216.52
Arthur Gretton53638226.18
Le Song62437159.27
Dale Schuurmans72760317.49