Paper Info
Title
Implementation and evaluation of a constraint-based multimodal fusion system for speech and 3D pointing gestures
Abstract
This paper presents an architecture for fusion of multimodal input streams for natural interaction with a humanoid robot as well as results from a user study with our system. The presented fusion architecture consists of an application independent parser of input events, and application specific rules. In the presented user study, people could interact with a robot in a kitchen scenario, using speech and gesture input. In the study, we could observe that our fusion approach is very tolerant against falsely detected pointing gestures. This is because we use speech as the main modality and pointing gestures mainly for disambiguation of objects. In the paper we also report about the temporal correlation of speech and gesture events as observed in the user study.
Year
DOI
Venue
2004
10.1145/1027933.1027964
ICMI
Keywords
Field
DocType
fusion architecture,user study,humanoid robot,application specific rule,multimodal input stream,gesture input,fusion approach,gesture event,input event,constraint-based multimodal fusion system,application independent parser,natural language,vision,gesture,speech,human factors
Application specific,Computer science,Gesture,Human–computer interaction,Artificial intelligence,Humanoid robot,Computer vision,Architecture,Speech recognition,Natural language,Parsing,Robot,Fusion system
Conference
ISBN
Citations 
PageRank 
1-58113-995-0
43
2.47
References 
Authors
19
3
Name
Order
Citations
PageRank
Hartwig Holzapfel116412.24
Kai Nickel261037.67
Rainer Stiefelhagen33512274.86