Explainable Artificial Intelligence And User Experience : Measuring the Effectiveness of a Communication Interface

Abstract

A recent focus in the field of artificial intelligence (AI) has been revolving around generative AI and deep neural networks utilized by them. This technology is often opaque by its nature, with AIs utilizing them often referred to as black boxes. Explainable artificial intelligence (xAI) is being actively researched, in an effort to add transparency to the AI algorithms. However, different kinds of xAI methods can produce immense amounts of data, which may not help a normal user in any way. This thesis presents a chat-based communication interface attached to an AI framework made for the videogame Super Mario Bros. The chat interface was used for transferring information from the AI agents playing the game to the user, as well as for a chatbot reacting to the game state. A survey was made to find out how the chat features affected the way the user or observer evaluated the intelligence of the AI. A total of 65 answers were collected. The results show that including the chat features lowered the average confidence of the responders, especially the ones who were very familiar with Super Mario Bros. The inclusion of the chat interface did not seem to affect the evaluations by much, otherwise. Importantly, the more time the responder spent on reviewing the AI and the chat interface, the more accurate their evaluations started to be, as the AIs presented in the survey were fake.