Paper Info
Title
QSAnglyzer: Visual Analytics for Prismatic Analysis of Question Answering System Evaluations
Abstract
Developing sophisticated artificial intelligence (AI) systems requires AI researchers to experiment with different designs and analyze results from evaluations (we refer this task as evaluation analysis). In this paper, we tackle the challenges of evaluation analysis in the domain of question-answering (QA) systems. Through in-depth studies with QA researchers, we identify tasks and goals of evaluation analysis and derive a set of design rationales, based on which we propose a novel approach termed prismatic analysis. Prismatic analysis examines data through multiple ways of categorization (referred as angles). Categories in each angle are measured by aggregate metrics to enable diverse comparison scenarios. To facilitate prismatic analysis of QA evaluations, we design and implement the Question Space Anglyzer (QSAnglyzer), a visual analytics (VA) tool. In QSAnglyzer, the high-dimensional space formed by questions is divided into categories based on several angles (e.g., topic and question type). Each category is aggregated by accuracy, the number of questions, and accuracy variance across evaluations. QSAnglyzer visualizes these angles so that QA researchers can examine and compare evaluations from various aspects both individually and collectively. Furthermore, QA researchers filter questions based on any angle by clicking to construct complex queries. We validate QSAnglyzer through controlled experiments and by expert reviews. The results indicate that when using QSAnglyzer, users perform analysis tasks faster (p <; 0.01) and more accurately (p <; 0.05), and are quick to gain new insight. We discuss how prismatic analysis and QSAnglyzer scaffold evaluation analysis, and provide directions for future research.
Year
DOI
Venue
2017
10.1109/VAST.2017.8585733
2017 IEEE Conference on Visual Analytics Science and Technology (VAST)
Keywords
Field
DocType
visual analytics,visualization,interactive visualization,question answering,multi-experiment analysis,visual comparison,visual exploration,prismatic analysis,H.5.2 [Information Interfaces and Presentation]: User Interfaces—
Categorization,Data mining,Question answering,Task analysis,Information retrieval,Computer science,Visual analytics,Scaffold Evaluation,Knowledge extraction
Conference
ISSN
ISBN
Citations 
2325-9442
978-1-5386-3164-5
0
PageRank 
References 
Authors
0.34
12
2
Name
Order
Citations
PageRank
Nan-Chen Chen1395.77
Been Kim235321.44