Full Text Available
Note: Clicking the button above will open the full text document at the original institutional repository in a new window.
Automated learning rates in machine learning for dynamic mini-batch sub-sampled losses
Thesis (PhD (Mechanical Engineering))--University of Pretoria, 2020.
Saved in:
| Other Authors: | |
|---|---|
| Format: | Thesis |
| Language: | English |
| Published: |
University of Pretoria
2024
|
| Subjects: | |
| Tags: |
No Tags, Be the first to tag this record!
|
| _version_ | 1867613552286629888 |
|---|---|
| access_status_str | Open Access |
| author2 | Wilke, Daniel Nicolas |
| author_browse | Wilke, Daniel Nicolas |
| author_facet | Wilke, Daniel Nicolas |
| collection | Thesis |
| dc_rights_str_mv | © 2021 University of Pretoria. All rights reserved. The copyright in this work vests in the University of Pretoria. No part of this work may be reproduced or transmitted in any form or by any means, without the prior written permission of the University of Pretoria. |
| description | Thesis (PhD (Mechanical Engineering))--University of Pretoria, 2020. |
| format | Thesis |
| id | oai:repository.up.ac.za:2263/95641 |
| institution | University of Pretoria (South Africa) |
| language | English |
| last_indexed | 2026-06-10T12:37:57.427Z |
| license_str | Other — see source repository |
| provenance_str_mv | Harvested via OAI-PMH from UPSpace — University of Pretoria Institutional Repository |
| publishDate | 2024 |
| publishDateRange | 2024 |
| publishDateSort | 2024 |
| publisher | University of Pretoria |
| publisherStr | University of Pretoria |
| record_format | dspace |
| source_str | UPSpace — University of Pretoria Institutional Repository |
| spelling | oai:repository.up.ac.za:2263/95641 Automated learning rates in machine learning for dynamic mini-batch sub-sampled losses Wilke, Daniel Nicolas u11207312@tuks.co.za Kafka, Dominic UCTD Machine learning Automated learning rates Machine learning Dynamic mini-batch Sub-sampled losses Engineering, built environment and information technology theses SDG-04 SDG-04: Quality education Engineering, built environment and information technology theses SDG-09 SDG-09: Industry, innovation and infrastructure Engineering, built environment and information technology theses SDG-12 SDG-12: Responsible consumption and production Thesis (PhD (Mechanical Engineering))--University of Pretoria, 2020. Learning rate schedule parameters remain some of the most sensitive hyperparameters in machine learning, as well as being challenging to resolve, in particular when mini-batch subsampling is considered. Mini-batch sub-sampling (MBSS) can be conducted in a number of ways, each with their own implications on the smoothness and continuity of the underlying loss function. In this study, dynamic MBSS, often applied in approximate optimization, is considered for neural network training. For dynamic MBSS, the mini-batch is updated for every function and gradient evaluation of the loss and gradient functions. The implication is that the sampling error between mini-batches changes abruptly, resulting in non-smooth and discontinuous loss functions. This study proposes an approach to automatically resolve learning rates for dynamic MBSS loss functions using gradient-only line searches (GOLS) over fifteen orders of magnitude. A systematic study is performed, which investigates the characteristics and the influence of training algorithms, neural network architectures and activation functions on the ability of GOLS to resolve learning rates. GOLS are shown to compare favourably against the state-ofthe-art probabilistic line search for dynamic MBSS loss functions. Matlab and PyTorch 1.0 implementations of GOLS are available for both practical training of neural networks as well as a research tool to investigate dynamic MBSS loss functions. mi2025 Mechanical and Aeronautical Engineering PhD (Mechanical Engineering) Unrestricted SDG-04: Quality education SDG-09: Industry, innovation and infrastructure SDG-12: Responsible consumption and production 2024-04-18T09:17:34Z 2024-04-18T09:17:34Z 2021 2020-12 Thesis * S2021 http://hdl.handle.net/2263/95641 en © 2021 University of Pretoria. All rights reserved. The copyright in this work vests in the University of Pretoria. No part of this work may be reproduced or transmitted in any form or by any means, without the prior written permission of the University of Pretoria. application/pdf University of Pretoria |
| spellingShingle | UCTD Machine learning Automated learning rates Machine learning Dynamic mini-batch Sub-sampled losses Engineering, built environment and information technology theses SDG-04 SDG-04: Quality education Engineering, built environment and information technology theses SDG-09 SDG-09: Industry, innovation and infrastructure Engineering, built environment and information technology theses SDG-12 SDG-12: Responsible consumption and production Automated learning rates in machine learning for dynamic mini-batch sub-sampled losses |
| title | Automated learning rates in machine learning for dynamic mini-batch sub-sampled losses |
| title_full | Automated learning rates in machine learning for dynamic mini-batch sub-sampled losses |
| title_fullStr | Automated learning rates in machine learning for dynamic mini-batch sub-sampled losses |
| title_full_unstemmed | Automated learning rates in machine learning for dynamic mini-batch sub-sampled losses |
| title_short | Automated learning rates in machine learning for dynamic mini-batch sub-sampled losses |
| title_sort | automated learning rates in machine learning for dynamic mini batch sub sampled losses |
| topic | UCTD Machine learning Automated learning rates Machine learning Dynamic mini-batch Sub-sampled losses Engineering, built environment and information technology theses SDG-04 SDG-04: Quality education Engineering, built environment and information technology theses SDG-09 SDG-09: Industry, innovation and infrastructure Engineering, built environment and information technology theses SDG-12 SDG-12: Responsible consumption and production |
| url | http://hdl.handle.net/2263/95641 |