City Research Online

Factors for the Generalisation of Identity Relations by Neural Networks

Kopparti, R. M. and Weyde, T. ORCID: 0000-0001-8028-9905 (2019). Factors for the Generalisation of Identity Relations by Neural Networks. Paper presented at the Thirty-sixth International Conference on Machine Learning (ICML 2019 ), 9-15 Jun 2019, Long Beach, USA.

Abstract

Many researchers implicitly assume that neural networks learn relations and generalise them to new unseen data. It has been shown recently, however, that the generalisation of feed-forward networks fails for identity relations.The proposed solution for this problem is to create an inductive bias with Differential Rectifier (DR) units. In this work we explore various factors in the neural network architecture and learning process whether they make a difference to the generalisation on equality detection of Neural Networks without and and with DR units in early and mid fusion architectures.

We find in experiments with synthetic data effects of the number of hidden layers, the activation function and the data representation. The training set size in relation to the total possible set of vectors also makes a difference. However, the accuracy never exceeds 61% without DR units at 50% chance level. DR units improve generalisation in all tasks and lead to almost perfect test accuracy in the Mid Fusion setting. Thus, DR units seem to be a promising approach for creating generalisation abilities that standard networks lack.

Publication Type: Conference or Workshop Item (Paper)
Additional Information: ICML 2019 Workshop on Understanding and Improving Generalization in Deep Learning, Long Beach, California, 2019
Subjects: Q Science > QA Mathematics > QA75 Electronic computers. Computer science
R Medicine > RC Internal medicine > RC0321 Neuroscience. Biological psychiatry. Neuropsychiatry
Departments: School of Mathematics, Computer Science & Engineering > Computer Science
Date Deposited: 09 Mar 2020 10:53
URI: https://openaccess.city.ac.uk/id/eprint/23875
[img]
Preview
Text - Accepted Version
Download (245kB) | Preview

Export

Downloads

Downloads per month over past year

View more statistics

Actions (login required)

Admin Login Admin Login