City Research Online

Abstract

Scenario-based testing has emerged as a common method for autonomous vehicles (AVs) safety assessment, offering a more efficient alternative to mile-based testing by focusing on high-risk scenarios. However, fundamental questions persist regarding its stopping rules, residual risk estimation, debug effectiveness, and the impact of simulation fidelity on safety claims. This paper argues that a rigorous statistical foundation is essential to address these challenges and enable rigorous safety assurance. By drawing parallels between AV testing and established software testing methods, we identify shared research gaps and reusable solutions. We propose proof-of-concept models to quantify the probability of failure per scenario (pfs) and evaluate testing effectiveness under varying conditions. Our analysis reveals that neither scenario-based nor mile-based testing universally outperforms the other. Furthermore, we give an example of formal reasoning about alignment of synthetic and real-world testing outcomes, a first step towards supporting statistically defensible simulation-based safety claims.

Publication Type:	Conference or Workshop Item (Paper)
Additional Information:	© 2026 IEEE. This accepted manuscript is made available under the terms of the Creative Commons Attribution License (CC-BY), which permits unrestricted use, distribution and reproduction in any medium, provided the original work is properly cited.
Subjects:	H Social Sciences > HN Social history and conditions. Social problems. Social reform Q Science > QA Mathematics > QA75 Electronic computers. Computer science T Technology > TL Motor vehicles. Aeronautics. Astronautics
Departments:	School of Science & Technology School of Science & Technology > Department of Computer Science School of Science & Technology > Department of Computer Science > Software Reliability
SWORD Depositor:	Symplectic Administrator

Export

Downloads