Personalized Chatbot Trustworthiness Ratings

Authors:
Publisher website
Google Scholar
Abstract
In this paper, we address a setting where a conversation agent, also know as a chatbot, cannot be modified and its training data cannot be accessed, and yet a neutral party wants to assess and communicate its trustworthiness to a user in a way that is tailored to the user’s priorities over the various trust issues (such as bias, abusive language, information leakage, or inappropriate communication complexity). Such a rating can help users choose among alternative chatbots, developers test their systems, business leaders price their offerings, and regulators set policies. We describe a chatbot rating methodology that relies on separate rating modules for each trust issue, and on users’ priority orderings among the relevant trust issues, to generate an aggregate personalized rating for the trustworthiness of a chatbot. The method is independent of the specific trust issues and is parametric to the aggregation procedure, thereby allowing for seamless generalization. We illustrate its general use, integrate it with a live chatbot, and evaluate it on four dialog datasets and representative user profiles, validated with a user survey.

This publication has 18 references indexed in Scilit: