Paper Info
Title
VisTA: Integrating Machine Intelligence with Visualization to Support the Investigation of Think-Aloud Sessions.
Abstract
Think-aloud protocols are widely used by user experience (UX) practitioners in usability testing to uncover issues in user interface design. It is often arduous to analyze large amounts of recorded think-aloud sessions and few UX practitioners have an opportunity to get a second perspective during their analysis due to time and resource constraints. Inspired by the recent research that shows subtle verbalization and speech patterns tend to occur when users encounter usability problems, we take the first step to design and evaluate an intelligent visual analytics tool that leverages such patterns to identify usability problem encounters and present them to UX practitioners to assist their analysis. We first conducted and recorded think-aloud sessions, and then extracted textual and acoustic features from the recordings and trained machine learning (ML) models to detect problem encounters. Next, we iteratively designed and developed a visual analytics tool, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">VisTA</italic> , which enables dynamic investigation of think-aloud sessions with a timeline visualization of ML predictions and input features. We conducted a between-subjects laboratory study to compare three conditions, i.e., <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">VisTA, VisTASimple</italic> (no visualization of the ML's input features), and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Baseline</italic> (no ML information at all), with 30 UX professionals. The findings show that UX professionals identified more problem encounters when using VisTA than Baseline by leveraging the problem visualization as an overview, anticipations, and anchors as well as the feature visualization as a means to understand what ML considers and omits. Our findings also provide insights into how they treated ML, dealt with (dis)agreement with ML, and reviewed the videos (i.e., play, pause, and rewind).
Year
DOI
Venue
2020
10.1109/TVCG.2019.2934797
IEEE transactions on visualization and computer graphics
Keywords
Field
DocType
Usability,Visual analytics,Tools,Machine intelligence,Feature extraction,Machine learning
Visualization,Computer science,Theoretical computer science,Human–computer interaction,Think aloud protocol
Journal
Volume
Issue
ISSN
26
1
1077-2626
Citations 
PageRank 
References 
3
0.37
18
Authors
6
Name
Order
Citations
PageRank
Mingming Fan193.46
Ke Wu230.37
Jian Zhao340526.77
Yue Li4151.34
Winter Wei551.47
Khai N. Truong62002162.82