Paper Info
Title
From the Curtain to Kansas: Conducting Wizard-of-Oz Studies in the Wild
Abstract
ABSTRACTConversational agents have been touted for their potential to support individuals over time as health coaches or personal assistants, but have yet to live up to this potential. Wizard-of-oz (WOz) methods enable researchers to test early prototypes of conversational applications before they are fully implemented, with a human “wizard” filling in the gaps in functionality. Current WOz methods, however, are more commonly used for studies in a lab setting, rather than deployment studies, which more accurately capture users’ interactions in-the-wild. We argue for the need for WOz methods for deployment studies that address key challenges, namely the need for easy-to-prototype technology that works reliably in the wild, as well as the user's expectations for 24/7 availability. We describe an initial approach that begins to address these challenges, as well as the insights gleaned from a two-week WOz study of t2.coach, a conversational agent health coach for diabetes self-management. We argue that the findings from our WOz study could not have been identified from a lab usability study with the same prototype, and the need for the research community to further develop methods for WOz deployment studies.
Year
DOI
Venue
2021
10.1145/3411763.3443446
Conference on Human Factors in Computing Systems
Keywords
DocType
Citations 
wizard-of-oz, conversational agents, chatbots, deployment study
Conference
0
PageRank 
References 
Authors
0.34
0
2
Name
Order
Citations
PageRank
Elliot G. Mitchell145.88
Lena Mamykina2112498.05