Paper Info
Title
Visual Query Answering By Entity-Attribute Graph Matching And Reasoning
Abstract
Visual Query Answering (VQA) is of great significance in offering people convenience: one can raise a question for details of objects, or high-level understanding about the scene, over an image. This paper proposes a novel method to address the VQA problem. In contrast to prior works, our method that targets single scene VQA, replies on graph-based techniques and involves reasoning. In a nutshell, our approach is centered on three graphs. The first graph, referred to as inference graph G(I) , is constructed via learning over labeled data. The other two graphs, referred to as query graph Q and entity-attribute graph G(EA), are generated from natural language query Q(nl) and image Img, that are issued from users, respectively. As GEA often does not take sufficient information to answer Q, we develop techniques to infer missing information of G(EA) with G(I) . Based on G(EA) and Q, we provide techniques to find matches of Q in G(EA), as the answer of Q(nl) in Img. Unlike commonly used VQA methods that are based on end-to-end neural networks, our graph-based method shows well-designed reasoning capability, and thus is highly interpretable. We also create a dataset on soccer match (Soccer-VQA)(1) with rich annotations. The experimental results show that our approach outperforms the state-of-the-art method and has high potential for future investigation.
Year
DOI
Venue
2019
10.1109/CVPR.2019.00855
2019 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2019)
Field
DocType
Volume
Graph,Visual query,Information retrieval,Computer science,Inference,Matching (graph theory),Natural language user interface,Artificial intelligence,Labeled data,Artificial neural network,Machine learning
Journal
abs/1903.06994
ISSN
Citations 
PageRank 
1063-6919
3
0.36
References 
Authors
23
5
Name
Order
Citations
PageRank
Peixi Xiong141.05
Huayi Zhan243.08
xin wang31857.35
Baivab Sinha430.70
Ying Wu54266246.00