City Research Online

Loss-size and Reliability Trade-offs Amongst Diverse Redundant Binary Classifiers

Salako, K. ORCID: 0000-0003-0394-7833 (2020). Loss-size and Reliability Trade-offs Amongst Diverse Redundant Binary Classifiers. In: Quantitative Evaluation of Systems 2020. 17th International Conference on the Quantitative Evaluation of SysTems (QEST 2020), 31 Aug - 3 Sep 2020, Online. doi: 10.1007/978-3-030-59854-9


Many applications involve the use of binary classifiers, including applications where safety and security are critical. The quantitative assessment of such classifiers typically involves receiver operator characteristic (ROC) methods and the estimation of sensitivity/specificity. But such techniques have their limitations. For safety/security critical applications, more relevant measures of reliability and risk should be estimated. Moreover, ROC techniques do not explicitly account for: 1) inherent uncertainties one faces during assessments, 2) reliability evidence other than the observed failure behaviour of the classifier, and 3) how this observed failure behaviour alters one's uncertainty about classifier reliability. We address these limitations using conservative Bayesian inference (CBI) methods, producing statistically principled, conservative values for risk/reliability measures of interest. Our analyses reveals trade-offs amongst all binary classifiers with the same expected loss { the most reliable classifiers are those most likely to experience high impact failures. This trade-off is harnessed by using diverse redundant binary classifiers.

Publication Type: Conference or Workshop Item (Paper)
Additional Information: The final authenticated version is available online at
Publisher Keywords: reliability assessment, binary classification, diverse redundancy, conservative Bayesian inference
Subjects: Q Science > QA Mathematics
Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Departments: School of Science & Technology > Computer Science > Software Reliability
[thumbnail of QEST2020_CameraReady.pdf]
Text - Accepted Version
Download (8MB) | Preview


Add to AnyAdd to TwitterAdd to FacebookAdd to LinkedinAdd to PinterestAdd to Email


Downloads per month over past year

View more statistics

Actions (login required)

Admin Login Admin Login