Towards Effective Conversational Agents:A Prototype-Based Approach for Facilitating Their Evaluation and Improvement

Using conversational agents (CAs) has become increasingly popular for organizations for various applications, such as customer service and healthcare. However, user satisfaction and engagement with CA adoption stay behind expectations. In this context, quality criteria (e.g., regarding dialog flow or representation) can serve as a basis to evaluate CAs and improve their effectiveness to consequently affect user satisfaction and engagement. Hereby, we contribute to emerging research of quality criteria by proposing a prototype-based approach to facilitate the evaluation and improvement of CAs design and effectiveness. Our approach involves deriving scenarios from synthesized CA criteria from preliminary work, creating prototypes, and evaluating them in expert interviews. By comparing prototypes against each other, the influence on CA effectiveness can be measured. Our results demonstrate an approach for applying the criteria and offer a promising direction towards designing, developing, and operating CAs.