Paper Info
Title
Analytic Quality: Evaluation of Performance and Insight in Multimedia Collection Analysis
Abstract
In this paper, we present analytic quality (AQ), a novel paradigm for the design and evaluation of multimedia analysis methods. AQ complements the existing evaluation methods based on either machine-driven benchmarks or user studies. AQ includes the notion of user insight gain and the time needed to acquire it, both critical aspects of large-scale multimedia collections analysis. To incorporate insight, AQ introduces a novel user model. In this model, each simulated user, or artificial actor, builds its insight over time, at any time operating with multiple categories of relevance. The methods are evaluated in timed sessions. The artificial actors interact with each method and steer the course by indicating relevant items throughout the session. AQ measures not only precision and recall, but also throughput, diversity of the results, and the accuracy of estimating the percentage of relevant items in the collection. AQ is shown to provide a wide picture of analytic capabilities of the evaluated methods and enumerate how their strengths differ for different purposes. The AQ time plots provide design suggestions for improving the evaluated methods. AQ is demonstrated to be more insightful than the classic benchmark evaluation paradigm both in terms of method comparison and suggestions for further design.
Year
DOI
Venue
2015
10.1145/2733373.2806279
ACM Multimedia
Keywords
Field
DocType
Evaluation,multimedia search and exploration,interactivity,user insight
Interactivity,Computer science,Precision and recall,User modeling,Throughput,Multimedia,User studies
Conference
Citations 
PageRank 
References 
3
0.41
30
Authors
3
Name
Order
Citations
PageRank
Jan Zahálka1348.80
Stevan Rudinac218820.45
Marcel Worring36439384.88