City Research Online

Weight Priors for Learning Identity Relations

Kopparti, R. M. and Weyde, T. ORCID: 0000-0001-8028-9905 (2019). Weight Priors for Learning Identity Relations. Paper presented at the Thirty-third Conference on Neural Information Processing Systems (NeurIPS 2019), 8-14 Dec 2019, Vancouver, Canada.

Abstract

Learning abstract and systematic relations has been an open issue in neural network learning for over 30 years. It has been shown recently that neural networks do not learn relations based on identity and are unable to generalize well to unseen data. The Relation Based Pattern (RBP) approach has been proposed as a solution for this problem. In this work, we extend RBP by realizing it as a Bayesian prior on network weights to model the identity relations. This weight prior leads to a modified regularization term in otherwise standard network learning. In our experiments, we show that the Bayesian weight priors lead to perfect generalization when learning identity based relations and do not impede general neural network learning. We believe that the approach of creating an inductive bias with weight priors can be extended easily to other forms of relations and will be beneficial for many other learning tasks.

Publication Type: Conference or Workshop Item (Paper)
Subjects: Q Science > QA Mathematics > QA75 Electronic computers. Computer science
R Medicine > RC Internal medicine > RC0321 Neuroscience. Biological psychiatry. Neuropsychiatry
Departments: School of Mathematics, Computer Science & Engineering > Computer Science
Date Deposited: 09 Mar 2020 10:58
URI: https://openaccess.city.ac.uk/id/eprint/23881
[img]
Preview
Text - Accepted Version
Download (292kB) | Preview

Export

Downloads

Downloads per month over past year

View more statistics

Actions (login required)

Admin Login Admin Login