City Research Online

Experiments in Learning Dyck-1 Languages with Recurrent Neural Networks

El-Naggar, N., Madhyastha, P. ORCID: 0000-0002-4438-8161 & Weyde, T. (2022). Experiments in Learning Dyck-1 Languages with Recurrent Neural Networks. In: Proceedings of the 3rd Human-Like Computing Workshop. Human-Like Computing Workshop (HLC 2022), 28-30 Sep 2022, Windsor, United Kingdom.


Considerable work, both theoretical and empirical, has shown that Recurrent Neural Network (RNN) architectures are capable of learning formal languages under specific conditions. In this study, we investigate the ability of linear and ReLU RNNs to learn Dyck-1 languages in whole sequence classification tasks. We observe that counting bracket sequences is learned but performance on full Dyck-1 recognition is poor. Models for both tasks do not generalise well to longer sequences. We determine correct weights for the given tasks with suitable architectures, but the standard setup for classification surprisingly departs from the correct values. We propose a regression setup with clipping that we find to stabilise correct weights, but it makes learning from random weight initialisation even less effective. Our observations suggest that Dyck-1 languages seem unlikely to be learned by ReLU RNNs for most practical applications.

Publication Type: Conference or Workshop Item (Paper)
Additional Information: © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
Publisher Keywords: Dyck-1 languages, Formal language learning, Generalisation, Classification, Systematicity
Subjects: Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Departments: School of Science & Technology > Computer Science
[thumbnail of El-Naggar.PP5.pdf]
Text - Published Version
Available under License Creative Commons Attribution.

Download (244kB) | Preview


Add to AnyAdd to TwitterAdd to FacebookAdd to LinkedinAdd to PinterestAdd to Email


Downloads per month over past year

View more statistics

Actions (login required)

Admin Login Admin Login