Paper Info
Title
A nonparametric Bayesian approach to learning multimodal interaction management
Abstract
Managing multimodal interactions between humans and computer systems requires a combination of state estimation based on multiple observation streams, and optimisation of time-dependent action selection. Previous work using partially observable Markov decision processes (POMDPs) for multimodal interaction has focused on simple turn-based systems. However, state persistence and implicit state transitions are frequent in real-world multimodal interactions. These phenomena cannot be fully modelled using turn-based systems, where the timing of system actions is a non-trivial issue. In addition, in prior work the POMDP parameterisation has been either hand-coded or learned from labelled data, which requires significant domain-specific knowledge and is labor-consuming. We therefore propose a nonparametric Bayesian method to automatically infer the (distributional) representations of POMDP states for multimodal interactive systems, without using any domain knowledge. We develop an extended version of the infinite POMDP method, to better address state persistence, implicit transition, and timing issues observed in real data. The main contribution is a “sticky” infinite POMDP model that is biased towards self-transitions. The performance of the proposed unsupervised approach is evaluated based on both artificially synthesised data and a manually transcribed and annotated human-human interaction corpus. We show statistically significant improvements (e.g. in ability of the planner to recall human bartender actions) over a supervised POMDP method.
Year
DOI
Venue
2012
10.1109/SLT.2012.6424162
SLT
Keywords
Field
DocType
artificially synthesised data,multimodal interaction,nonparametric bayesian approach,user interfaces,domain-specific knowledge,bayes methods,multiple observation streams,time-dependent action selection,implicit state transitions,unsupervised approach,multimodal interaction management learning,pomdp,timing issues,hdp,computer systems,supervised pomdp method,turn-based systems,decision theory,real-world multimodal interactions,state persistence,partially observable markov decision processes,markov processes,human bartender actions,unsupervised learning,annotated human-human interaction corpus
Markov process,Computer science,Markov decision process,Unsupervised learning,Decision theory,Artificial intelligence,Multimodal interaction,Domain knowledge,Pattern recognition,Partially observable Markov decision process,Speech recognition,Action selection,Machine learning
Conference
ISSN
ISBN
Citations 
2639-5479
978-1-4673-5124-9
2
PageRank 
References 
Authors
0.36
9
2
Name
Order
Citations
PageRank
Zhuoran Wang18810.98
Oliver Lemon2107286.38