City Research Online

Abstract

Dependability requirements for some systems are so stringent that sufficient assurance of their satisfaction cannot be achieved by evidence of successful operation before deployment. The dominant concern is often that critical design faults may still be present when the system is deployed. To gain regulatory approval to operate such system, a convincing demonstration must be produced that accidents will be as unlikely as required. Yet experience shows that every now and then such a claim, despite the complex process in place to ensure it is correct, is proved wrong in operation. The Boeing 737 MAX is just one recent, striking example.

We contend that the practice of risk assessment needs to take into account the inevitable doubt that affects any claim of extreme safety. We first outline how this doubt affects the bounds one can reasonably claim for probability of accidents. During early operation, this “reasonable” bound is much higher than the formal claim accepted by regulators, and depends heavily on the probability of that accepted claim being wrong. But this reasonable estimate then improves over time, if the system does operate without accidents or other surprises. We thus outline an argument that gives a more solid basis to current practices for authorising early operation of critical systems. We then show how evidence supporting “fall-back” arguments for even modest levels of safety can improve the bounds that can be claimed during early operation.

Last, we discuss possible improvements to the risk assessment processes, and research directions to address and mitigate the impact of doubt on a system safety justification.

Publication Type:	Conference or Workshop Item (Paper)
Additional Information:	This version of the contribution has been accepted for publication, after peer review but is not the Version of Record and does not reflect post-acceptance improvements, or any corrections. The Version of Record is available online at: https://doi.org/10.1007/978-3-032-02018-5_17. Use of this Accepted Version is subject to the publisher’s Accepted Manuscript terms of use https://www.springernature.com/gp/open-research/policies/accepted-manuscript-terms.
Publisher Keywords:	Ultra-high dependability, Epistemic uncertainty, Quantitative risk bounds, Assurances cases, Fall back safety claims
Subjects:	H Social Sciences > HD Industries. Land use. Labor Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Departments:	School of Science & Technology School of Science & Technology > Department of Computer Science School of Science & Technology > Department of Computer Science > Software Reliability
SWORD Depositor:	Symplectic Administrator

Export

Downloads