Full Text Available
Note: Clicking the button above will open the full text document at the original institutional repository in a new window.
Initialisation of noise-regularised neural networks
Thesis (MSc)--Stellenbosch University, 2021.
Saved in:
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis |
| Language: | en_ZA |
| Published: |
Stellenbosch : Stellenbosch University
2021
|
| Subjects: | |
| Tags: |
No Tags, Be the first to tag this record!
|
| _version_ | 1867613954054815744 |
|---|---|
| access_status_str | Open Access |
| author | Van Biljon, Elan |
| author2 | Kroon, R. S. (Steve) |
| author_browse | Kroon, R. S. (Steve) Van Biljon, Elan |
| author_facet | Kroon, R. S. (Steve) Van Biljon, Elan |
| author_sort | Van Biljon, Elan |
| collection | Thesis |
| dc_rights_str_mv | Stellenbosch University |
| description | Thesis (MSc)--Stellenbosch University, 2021. |
| format | Thesis |
| id | oai:scholar.sun.ac.za:10019.1/123661 |
| institution | Stellenbosch University (South Africa) |
| language | en_ZA |
| last_indexed | 2026-06-10T12:44:20.637Z |
| license_str | Other — see source repository |
| provenance_str_mv | Harvested via OAI-PMH from SUNScholar — Stellenbosch University Repository |
| publishDate | 2021 |
| publishDateRange | 2021 |
| publishDateSort | 2021 |
| publisher | Stellenbosch : Stellenbosch University |
| publisherStr | Stellenbosch : Stellenbosch University |
| record_format | dspace |
| source_str | SUNScholar — Stellenbosch University Repository |
| spelling | oai:scholar.sun.ac.za:10019.1/123661 Initialisation of noise-regularised neural networks Van Biljon, Elan Kroon, R. S. (Steve) Stellenbosch University. Faculty of Science. Dept. of Mathematical Sciences. Division Computer Science. Deep learning (Machine learning) Neural networks (Computer science) -- Noise Stochastic regularisation Critical initialisation Signal propagation Neural network initialisation Noise (Computer science) UCTD Thesis (MSc)--Stellenbosch University, 2021. ENGLISH ABSTRACT: Recently, proper initialisation and stochastic regularisation techniques have greatly improved the performance and ease of training of neural networks. Some research has gone into how the magnitude of the initial weights impact optimisation, while others have focused on how initialisation affects signal propagation. In terms of noise regularisation, dropout has allowed networks to train relatively quickly and reduced overfitting. Much research has gone towards understanding why dropout improves the generalisation of networks. Two major theories are (i) that it prevents neurons from becoming too dependent on the output of other neurons and (ii) that dropout leads a network to optimise a smoother loss landscape. Despite this, our theoretical understanding of the interaction between regularisation and initialisation is sparse. Thus, the aim of this work was to broaden our knowledge of how initialisation and stochastic regularisation interact and what impact this has on network training and performance. Because rectifier activation functions are widely used, we extended new network signal propagation theory to rectifier networks that may use stochastic regularisation. Our theory predicted a critical initialisation that allows for stable pre-activation variance signal propagation. However, our theory also indicated that stochastic regularisation reduces the depth to which correlation information can propagate in ReLU networks. We validated this theory and showed that it accurately predicts a boundary across which networks do not train effectively. We then extended the investigation by conducting a large-scale randomised control trial to search for initialisations in a region that conserves input signal around the critical initialisation in the hopes of finding initialisations that provide advantages to training or generalisation. We compare the critical initialisation to 10 other initialisation schemes in a trial that consisted of over 12000 networks. We found that initialisations much larger than the critical initialisation provide extremely poor performance, while network initialisations close to the critical initialisation provide similar performance. No initialisations clearly outperformed the critical initialisation. Thus, we recommend it as a safe default for practitioners. AFRIKAANSE OPSOMMING: Geen opsomming beskikbaar. The financial assistance of the Council for Scientific and Industrial Research (CSIR) towards this research is hereby acknowledged. Opinions expressed and conclusions arrived at, are those of the author and are not necessarily to be attributed to the CSIR. Masters 2021-09-09T14:02:23Z 2021-12-22T14:14:36Z 2021-09-09T14:02:23Z 2021-12-22T14:14:36Z 2021-12 Thesis http://hdl.handle.net/10019.1/123661 en_ZA Stellenbosch University xiii, 146 pages : illustrations application/pdf Stellenbosch : Stellenbosch University |
| spellingShingle | Deep learning (Machine learning) Neural networks (Computer science) -- Noise Stochastic regularisation Critical initialisation Signal propagation Neural network initialisation Noise (Computer science) UCTD Van Biljon, Elan Initialisation of noise-regularised neural networks |
| title | Initialisation of noise-regularised neural networks |
| title_full | Initialisation of noise-regularised neural networks |
| title_fullStr | Initialisation of noise-regularised neural networks |
| title_full_unstemmed | Initialisation of noise-regularised neural networks |
| title_short | Initialisation of noise-regularised neural networks |
| title_sort | initialisation of noise regularised neural networks |
| topic | Deep learning (Machine learning) Neural networks (Computer science) -- Noise Stochastic regularisation Critical initialisation Signal propagation Neural network initialisation Noise (Computer science) UCTD |
| url | http://hdl.handle.net/10019.1/123661 |
| work_keys_str_mv | AT vanbiljonelan initialisationofnoiseregularisedneuralnetworks |