Paper Info
Title
QaldGen: Towards Microbenchmarking of Question Answering Systems over Knowledge Graphs
Abstract
Over the last years, a number of Knowledge Graph (KG) based Question Answering (QA) systems have been developed. Consequently, the series of Question Answering Over Linked Data (QALD1-QALD9) challenges and other datasets have been proposed to evaluate these systems. However, the QA datasets contain a fixed number of natural language questions and do not allow users to select micro benchmarking samples of the questions tailored towards specific use-cases. We propose QaldGen, a framework for microbenchmarking of QA systems over KGs which is able to select customised question samples from existing QA datasets. The framework is flexible enough to select question samples of varying sizes and according to the user-defined criteria on the most important features to be considered for QA benchmarking. This is achieved using different clustering algorithms. We compare state-of-the-art QA systems over knowledge graphs by using different QA benchmarking samples. The observed results show that specialised micro-benchmarking is important to pinpoint the limitations of the various QA systems and its components.
Year
DOI
Venue
2019
10.1007/978-3-030-30796-7_18
Lecture Notes in Computer Science
DocType
Volume
ISSN
Conference
11779
0302-9743
Citations 
PageRank 
References 
2
0.35
0
Authors
7
Name
Order
Citations
PageRank
Kuldeep Singh130.70
Muhammad Saleem219421.78
Abhishek Nadgeri321.70
Felix Conrads4102.70
J. Z. Pan5254.59
Axel-Cyrille Ngonga Ngomo61775139.40
Jens Lehmann75375355.08