Predicting and Exploring Abandonment Signals in a Banking Task-Oriented Chatbot Service

Chieh Hsu, Hsin Chien Tung, Hong Han Shuai, Yung-Ju Chang*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

Abstract

In this study, we developed predictive models to address the problem of chatbot abandonment, a problem that can result in losing business opportunities. Specifically, we target on a conversation log dataset of a banking chatbot involving 1,373 users and hand-crafted features. By leveraging a pre-trained BERT model on the textural features and the hand-crafted features, the model achieved an F1-score of 0.89 in predicting discontinued conversation and 0.80 in predicting abandonment. Our findings indicate that textual features help capture more abandonment, while hand-crafted features improve detection precision. Our analysis with SHAP and LIME revealed that user typing, the chatbot expressing inability of recognizing intent, and the chatbot asking what users want to do during an ongoing conversation are top signals of user abandoning the chatbot. These findings suggest that chatbot designers should consider providing pre-set options or constraints for user inputs and presenting possible intents to the user and avoid expressing inability, incompetence, or ignoring the users’ current attempt.

Original languageEnglish
JournalInternational Journal of Human-Computer Interaction
DOIs
StateAccepted/In press - 2023

Keywords

  • abandonment
  • BERT
  • conversation breakdown
  • explainable AI
  • machine learning
  • Task-oriented chatbot

Fingerprint

Dive into the research topics of 'Predicting and Exploring Abandonment Signals in a Banking Task-Oriented Chatbot Service'. Together they form a unique fingerprint.

Cite this