City Research Online

Abstract

Many cases of AI-induced errors and hazards may seem aptly characterized as "overreliance" on automation: people behave as though they complied too willingly with the "suggestions" or "decisions" of AI software. We argue that this terminology often misleads its users about the real causes of such errors, and may lead to ineffective or counterproductive precautions. We point at various reasons why even users who mistrust an AI system are seen to make the kind of mistakes that lead an observer to conjecture "overreliance". Cognitive, affective, organizational etc. mechanisms that cause apparent compliance with the AI's outputs may all coexist, requiring analysis of how their effects, and those of precautions against them, compound and interact.
We give examples of such alternative explanations for apparent excessive compliance with AI, and of the potential harm if the latter is only blamed on inappropriate user attitudes.
To avoid this risk we recommend applying two complementary viewpoints for analysis for such failures of human-AI systems: "diversity", the extent to which the user, in the human-AI system as implemented, can cope safely with situations in which the AI failed to do so (or vice versa); and "adaptation", the ways the presence of AI changes (and improves and/or degrades) how people operate; we highlight some factors that cause such changes to reduce AI-human diversity and thus simulate "overreliance".

Publication Type:	Conference or Workshop Item (Paper)
Additional Information:	This version of the article has been accepted for publication, after peer review (when applicable) and is subject to Springer Nature’s AM terms of use, but is not the Version of Record and does not reflect post-acceptance improvements, or any corrections. The Version of Record will be available online at: https://link.springer.com/series/558
Publisher Keywords:	Automation bias, Trust in AI, AI-human diversity, Rational adaptation
Subjects:	Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Departments:	School of Science & Technology School of Science & Technology > Department of Computer Science
SWORD Depositor:	Symplectic Administrator

Export

Downloads