City Research Online

Abstract

We outline the principles of classical assurance for computer-based systems that pose significant risks. We then consider application of these principles to systems that employ Artificial Intelligence (AI) and Machine Learning (ML).
On its own, testing is insufficient for assurance when very high levels of confidence are required. Hence, a key element in the “dependability” perspective is a requirement to have thorough understanding of the internal design and operation (and hence behavior) of critical system components and their interaction. This is considered infeasible for AI and ML because their internal operation is developed experimentally over a limited (albeit large) set of training examples and is opaque to detailed understanding. Hence the dependability perspective, as we apply it here, aims to minimize trust in AI and ML elements by using “defense in depth” with a hierarchy of less complex systems, some of which may be highly assured conventionally engineered components, to “guard” them. This may be contrasted with what we call the “trust-worthiness” perspective that seeks to apply assurance to the AI and ML elements themselves by various forms of careful training, fine tuning, internal “guardrails” and automated examination.
In cyber-physical and many other systems, it is difficult to provide guards that do not depend on AI and ML to perceive their environment (e.g., other vehicles sharing the road with a self-driving car), so both perspectives are needed and there is a continuum or spectrum between them. We focus on architectures toward the dependability end of the continuum and invite others to consider additional points along the spectrum.
For guards that require perception using AI and ML, we examine ways to minimize the trust placed in these elements; they include diversity, defense in depth, explanations, and micro-ODDs (Operational Design Domains). We also examine methods to enforce acceptable behavior, given a model of the world. These include classical cyber-physical calculations and envelopes, and normative rules based on overarching principles, constitutions, ethics, and reputation.
We apply our perspective to autonomous systems, AI systems for specific functions, general-purpose AI such as Large Language Models (LLMs), and Artificial General Intelligence (AGI), and we propose current best practice and conclude with a fourfold agenda for research in which we recommend development and application of: a) new methods for hazard analysis suited to AI systems; b) layered recursively structured architectures for runtime verification and defense in depth; c) assurance for AI-based perception, and d) improved understanding of human and machine cognition, shared intentionality, and emergent behavior.

Publication Type:	Report
Additional Information:	Published by SRI, Computer Science Laboratory. Available online at: https://www.csl.sri.com/~rushby/papers/aisafety24.pdf
Subjects:	Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Departments:	School of Science & Technology School of Science & Technology > Department of Computer Science
SWORD Depositor:	Symplectic Administrator

Export

Downloads