Trust, Confidence, and Uncertainty in AI Models

Trust, Confidence, and Uncertainty in AI Models Michael J. Elliott, José A.B. Fortes University of Florida Gainesville, FL, United States Even for highly accurate artificial intelligence (AI) models, the small possibility of costly mistakes leaves users wondering whether they can be trusted. One approach to gaining users’ trust is to be honest about the possibility of AI models making mistakes before they are committed, so that users may decide when to accept a model’s outputs or take an alternative course of action. This requires sharing with the user information that communicates uncertainty in the model’s abilities. If an AI model’s mistakes can be reliably anticipated, AI models can be trusted in specific instances without needing to be trustworthy in general. We conducted a cross-disciplinary survey of research on trust, confidence, and uncertainty in AI applications. So far, we have found many independent threads of research that are inherently complementary, yet difficult to reconcile due to conflicting language, problem formulations, and application specializations. To begin to connect these ideas, we propose a conceptual framework that formalizes the interactions between trust, confidence, and uncertainty. Following the work of Jacovi et al. (2021), we define these terms in the context of a contract. A contract defines what the user considers to be acceptable AI model behavior. Oftentimes, the user simply expects the model to produce a “correct” output, such as correctly predicting tomorrow’s weather or correctly biometric authentication. More generally, however, a contract may also impose other constraints and criteria, such as limiting costs or imposing safety requirements. Even if a model is able to solve the identified problem, doing so may violate its contract. An AI model that upholds a contract is said to be trustworthy to that contract. We define confidence as the user’s estimated probability that the model is trustworthy as a function of perceived uncertainty. A user trusts an AI model when they have high confidence that the model is trustworthy. What constitutes “high” confidence depends on the user’s willingness to risk mistakenly trusting the model when it is not actually trustworthy. Finally, uncertainty exists when available information gives the user reason to doubt that the model is trustworthy. We refer to methods of quantifying uncertainty – either directly or indirectly – as uncertainty measures. Using our conceptual framework, we organize the surveyed research into three topics: classifying and measuring uncertainty, transforming uncertainty information into a single confidence score, and evaluating how well confidence scores distinguish between instances where a model is trustworthy or untrustworthy. Taken together, these methods form the building blocks for self-aware AI systems, i.e., AIpowered systems that report confidence in one or more AI models at their disposal. Self-awareness is imperative for applications where it is not feasible for a human user to judge every instance of an AI model’s use, whether because of time constraints, problem complexity, or prohibitive costs. Furthermore, self-aware systems can be designed to be fully autonomous by deciding for themselves what actions to take according to automated trust rules. We ground our conceptual framework by considering how trust can be integrated into an autonomous edge-to-cloud AI system to decide where processing should occur. For this use case, we identify a combination of methods from various research areas to quantify uncertainty, model confidence, and decide when to trust a lightweight edge-based model or defer to a more reliable cloud-based model. Finally, we consider how our conceptual framework for trust, confidence, and uncertainty may be adapted to other use cases beyond deciding when to trust AI models. References [1] Alon Jacovi, Ana Marasović, Tim Miller, and Yoav Goldberg. 2021. Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT '21). Association for Computing Machinery, New York, NY, USA, 624–635. https://doi.org/10.1145/3442188.3445923