Engagement and Usability of Conversational Search – A Study of a Medical Resource Center Chatbot

Due to advances in natural language understanding, chat- bots have become popular for assisting users in various tasks, for example, searching. Chatbots allow natural-language queries, which can be useful in case of complex information needs, and they provide a higher level of interactivity by displaying information in a dialog-like format. However, chatbots are often only used as auxiliaries for a graphical search user interface (SUI). Thus, they must be engaging and usable so that users both want to and are able to use them. In this study, we conduct a controlled interactive information retrieval experiment follow- ing a within-subject design to compare a chatbot to a graphical SUI in terms of engagement and usability. Our findings point towards the need for flawless usability in order for conversational search interfaces to (1) be able to provide additional value in information retrieval tasks and (2) elicit a higher level of engagement compared to their SUI-based counterparts.