City Research Online

From explanation to intervention: Interactive knowledge extraction from Convolutional Neural Networks used in radiology

Ngan, K. H., Mansouri-Benssassi, E., Phelan, J. , Townsend, J. & Garcez, A. D. (2024). From explanation to intervention: Interactive knowledge extraction from Convolutional Neural Networks used in radiology. PLoS ONE, 19(4), article number e0293967. doi: 10.1371/journal.pone.0293967


Deep Learning models such as Convolutional Neural Networks (CNNs) are very effective at extracting complex image features from medical X-rays. However, the limited interpretability of CNNs has hampered their deployment in medical settings as they failed to gain trust among clinicians. In this work, we propose an interactive framework to allow clinicians to ask what-if questions and intervene in the decisions of a CNN, with the aim of increasing trust in the system. The framework translates a layer of a trained CNN into a measurable and compact set of symbolic rules. Expert interactions with visualizations of the rules promote the use of clinically-relevant CNN kernels and attach meaning to the rules. The definition and relevance of the kernels are supported by radiomics analyses and permutation evaluations, respectively. CNN kernels that do not have a clinically-meaningful interpretation are removed without affecting model performance. By allowing clinicians to evaluate the impact of adding or removing kernels from the rule set, our approach produces an interpretable refinement of the data-driven CNN in alignment with medical best practice.

Publication Type: Article
Additional Information: Copyright: © 2024 Ngan et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Publisher Keywords: Radiography, Radiology, Neural Networks, Computer
Subjects: Q Science > QA Mathematics > QA75 Electronic computers. Computer science
R Medicine > RC Internal medicine
Departments: School of Science & Technology
School of Science & Technology > Computer Science
SWORD Depositor:
[thumbnail of journal.pone.0293967.pdf]
Text - Published Version
Available under License Creative Commons Attribution.

Download (4MB) | Preview


Add to AnyAdd to TwitterAdd to FacebookAdd to LinkedinAdd to PinterestAdd to Email


Downloads per month over past year

View more statistics

Actions (login required)

Admin Login Admin Login