Paper Info
Title
Chatbot or Chat-Blocker: Predicting Chatbot Popularity before Deployment
Abstract
Chatbots are widely employed in various scenarios. However, given the high costs of chatbot development and chatbots' tremendous social influence, chatbot failures may inevitably lead to a huge economic loss. Previous chatbot evaluation frameworks rely heavily on human evaluation, lending little support for automatic early-stage chatbot examination prior to deployment. To reduce the risk of potential loss, we propose a computational approach to extracting features and training models that make a priori prediction about chatbots' popularity, which indicates chatbot general performance. The features we extract cover chatbot Intent, Conversation Flow, and Response Design. We studied 1050 customer service chatbots on one of the most popular chatbot service platforms. Our model achieves 77.36% prediction accuracy among very popular and very unpopular chatbots, making the first step towards computational feedback before chatbot deployment. Our evaluation results also reveal the key design features associated with chatbot popularity and offer guidance on chatbot design.
Year
DOI
Venue
2021
10.1145/3461778.3462147
PROCEEDINGS OF THE 2021 ACM DESIGNING INTERACTIVE SYSTEMS CONFERENCE (DIS 2021)
Keywords
DocType
Citations 
chatbot design, early-stage computational feedback, predicting chatbot popularity, identifying key design features
Conference
0
PageRank 
References 
Authors
0.34
0
4
Name
Order
Citations
PageRank
Mingkun Gao142.08
Xiaotong Liu29110.11
Anbang Xu335130.52
Rama Akkiraju457149.91