City Research Online

Abstract

The application of feed forward neural networks (FFNN) to tasks involving high-dimensional data has always presented problems which emanate from the simple fact that these networks can not be scaled up unreservedly without serious side effects.

Gradient descent optimisation methods, independently of how well they perform in lower dimensions, will reach their limitations as soon as the search space reaches a critical dimension. For certain kernels and training data, this should be expected to happen sooner rather than later because the volume of this space grows exponentially with the number of input variables - something also known as curse of dimensionality.

For complex problems, the optimisation process may be hindered further by the appearance of numerous local minima due to the increased complexity and multi-modality of the error surface. In addition, neurons saturate and lose sensitivity when an excessively high input signal is received. This results in information being blocked and training impeded.

Parallelisation of FFNN proves extremely difficult in practice due to the exuberant communication overheads. This problem, which relates to the fine rather than coarse-grain parallelism inherent in the FFNN topology, removes virtually any possibility for efficient parallel distributed processing. The prospect of winning over the curse of dimensionality remains, largely, utopian.

In this thesis, a methodology for replacing the monolithic FFNN with an entity of simpler and smaller FFNN units is proposed. Our motivation stems from the inability of the single FFNN to deal effectively with all the problems mentioned above. Furthermore, although existing neural network models, be they modular or monolithic, are relatively successful in addressing issues of generalisation, specialisation and confidence of prediction, the problems associated with high-dimensional data and scaling remain basically unanswered. The thesis that the brain is not only characterised by a massively connected network of neurons but also by the existence of different computational systems operating at different levels of abstraction and specialising at different functions is by itself a right justification to replace the single FFNN with the entities. The claim here is that the use of the entities not only eliminates the aforementioned scaling problems, hence, allowing for network implementations with, virtually, no size restrictions, but also improves generalisation ability and training consistency, favours a coarse-grain parallelisation of the training process and promotes a computational model which can be studied with an arbitrary level of abstraction.

The concept of neural network decomposition is materialised with the construction of three different FFNN entity models, namely, classes 1, 2 and 3. A mathematical proof that these models are universal function approximators is accomplished with the aid of the Stone-Weierstrass theorem.

Finally, the generalisation ability and training consistency of the entities as well as time benefits obtained by parallelising their training procedure, are assessed in practice. These empirical results support the claims about the benefits obtained from the use of the entities and the thesis that they can safely replace single FFNN in applications of prohibitively high dimensionality.

Publication Type:	Thesis (Doctoral)
Subjects:	Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Departments:	School of Science & Technology > Computer Science School of Science & Technology > School of Science & Technology Doctoral Theses Doctoral Theses

Export

Downloads