Tell Me When Users Leave: Predicting Users' Abandonment of A Task-Oriented Chatbot Service using Explainable Deep Learning

Task-oriented chatbots have been widely used by businesses to support users in accomplishing predefined tasks. Yet, conversation breakdowns could result in users abandoning the chatbot service. Detecting or early predicting signals of users' chatbot abandonment could help businesses know when to provide assistance. Based on an annotated conversation log involving 1,837 users, we built two models, one end-to-end model built on top of pre-trained BERT models, and the other being an attention-based deep learning model trained from 102 different handcrafted features derived from annotated messages. The former achieved an AUROC of 90%. The latter explainable model, despite the extra effort of adding annotations, achieved a higher AUROC of 95.7% and provided additional insights into important features indicative of service abandonment, such as input types, error types, and presence of users' information-request within recently exchanged messages.