Full Text Available
Note: Clicking the button above will open the full text document at the original institutional repository in a new window.
Scalable hierarchical evolution strategies
Hierarchical reinforcement learning (HRL) has been steadily growing in popularity for solving the hardest reinforcement learning problems. However, current HRL algorithms are relatively slow and brittle to hyperparameter changes. This paper offers a solution to these slow and brittle HRL algorithms,...
Saved in:
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis |
| Language: | English |
| Published: |
Department of Computer Science
2023
|
| Subjects: | |
| Tags: |
No Tags, Be the first to tag this record!
|
| _version_ | 1867613490267553792 |
|---|---|
| access_status_str | Open Access |
| author | Abramowitz, Sasha |
| author2 | Nitschke, Geoff |
| author_browse | Abramowitz, Sasha Nitschke, Geoff |
| author_facet | Nitschke, Geoff Abramowitz, Sasha |
| author_sort | Abramowitz, Sasha |
| collection | Thesis |
| description | Hierarchical reinforcement learning (HRL) has been steadily growing in popularity for solving the hardest reinforcement learning problems. However, current HRL algorithms are relatively slow and brittle to hyperparameter changes. This paper offers a solution to these slow and brittle HRL algorithms, by investigating a novel method combining Scalable Evolution Strategies (SES) and HRL. S-ES, named for its excellent scalability, was popularised by Open AI when they showed its performance to be comparable to state-of-the art policy gradient methods. However, S-ES has not been tested in conjunction with HRL methods, which empower temporal abstraction thus allowing agents to tackle more challenging problems. We introduce a novel method merging S-ES and HRL, which creates a highly scalable and fast (wall-clock time) algorithm. We demonstrate that S-ES needs no hyper-parameter tuning for the HRL tasks tested and is indifferent to delayed rewards. This results in a method that is significantly faster than gradient-based HRL methods while having competitive task performance. We extend this method using transfer learning with the aim of increasing task performance and novelty search with the goal of improving its exploration characteristics. The paper's main contribution is thus a novel evolutionary HRL method, namely Scalable Hierarchical Evolution Strategies, which yields greater learning speed and competitive task-performance compared to state-of-the-art gradient-based methods, across a range of tasks. |
| format | Thesis |
| id | oai:open.uct.ac.za:11427/36999 |
| institution | University of Cape Town (South Africa) |
| language | eng |
| last_indexed | 2026-06-10T12:36:58.641Z |
| license_str | Not specified — see source repository |
| provenance_str_mv | Harvested via OAI-PMH from UCTD — University of Cape Town Open Access Repository |
| publishDate | 2023 |
| publishDateRange | 2023 |
| publishDateSort | 2023 |
| publisher | Department of Computer Science |
| publisherStr | Department of Computer Science |
| record_format | dspace |
| source_str | UCTD — University of Cape Town Open Access Repository |
| spelling | oai:open.uct.ac.za:11427/36999 Scalable hierarchical evolution strategies Abramowitz, Sasha Nitschke, Geoff Computer Science Hierarchical reinforcement learning (HRL) has been steadily growing in popularity for solving the hardest reinforcement learning problems. However, current HRL algorithms are relatively slow and brittle to hyperparameter changes. This paper offers a solution to these slow and brittle HRL algorithms, by investigating a novel method combining Scalable Evolution Strategies (SES) and HRL. S-ES, named for its excellent scalability, was popularised by Open AI when they showed its performance to be comparable to state-of-the art policy gradient methods. However, S-ES has not been tested in conjunction with HRL methods, which empower temporal abstraction thus allowing agents to tackle more challenging problems. We introduce a novel method merging S-ES and HRL, which creates a highly scalable and fast (wall-clock time) algorithm. We demonstrate that S-ES needs no hyper-parameter tuning for the HRL tasks tested and is indifferent to delayed rewards. This results in a method that is significantly faster than gradient-based HRL methods while having competitive task performance. We extend this method using transfer learning with the aim of increasing task performance and novelty search with the goal of improving its exploration characteristics. The paper's main contribution is thus a novel evolutionary HRL method, namely Scalable Hierarchical Evolution Strategies, which yields greater learning speed and competitive task-performance compared to state-of-the-art gradient-based methods, across a range of tasks. 2023-02-23T08:43:16Z 2023-02-23T08:43:16Z 2022 2023-02-20T12:09:06Z Master Thesis Masters MSc http://hdl.handle.net/11427/36999 eng application/pdf Department of Computer Science Faculty of Science |
| spellingShingle | Computer Science Abramowitz, Sasha Scalable hierarchical evolution strategies |
| thesis_degree_str | Master's |
| title | Scalable hierarchical evolution strategies |
| title_full | Scalable hierarchical evolution strategies |
| title_fullStr | Scalable hierarchical evolution strategies |
| title_full_unstemmed | Scalable hierarchical evolution strategies |
| title_short | Scalable hierarchical evolution strategies |
| title_sort | scalable hierarchical evolution strategies |
| topic | Computer Science |
| url | http://hdl.handle.net/11427/36999 |
| work_keys_str_mv | AT abramowitzsasha scalablehierarchicalevolutionstrategies |