Full Text Available
Note: Clicking the button above will open the full text document at the original institutional repository in a new window.
Scalable hierarchical evolution strategies
Hierarchical reinforcement learning (HRL) has been steadily growing in popularity for solving the hardest reinforcement learning problems. However, current HRL algorithms are relatively slow and brittle to hyperparameter changes. This paper offers a solution to these slow and brittle HRL algorithms,...
Saved in:
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis |
| Language: | English |
| Published: |
Department of Computer Science
2023
|
| Subjects: | |
| Tags: |
No Tags, Be the first to tag this record!
|
| Summary: | Hierarchical reinforcement learning (HRL) has been steadily growing in popularity for solving the hardest reinforcement learning problems. However, current HRL algorithms are relatively slow and brittle to hyperparameter changes. This paper offers a solution to these slow and brittle HRL algorithms, by investigating a novel method combining Scalable Evolution Strategies (SES) and HRL. S-ES, named for its excellent scalability, was popularised by Open AI when they showed its performance to be comparable to state-of-the art policy gradient methods. However, S-ES has not been tested in conjunction with HRL methods, which empower temporal abstraction thus allowing agents to tackle more challenging problems. We introduce a novel method merging S-ES and HRL, which creates a highly scalable and fast (wall-clock time) algorithm. We demonstrate that S-ES needs no hyper-parameter tuning for the HRL tasks tested and is indifferent to delayed rewards. This results in a method that is significantly faster than gradient-based HRL methods while having competitive task performance. We extend this method using transfer learning with the aim of increasing task performance and novelty search with the goal of improving its exploration characteristics. The paper's main contribution is thus a novel evolutionary HRL method, namely Scalable Hierarchical Evolution Strategies, which yields greater learning speed and competitive task-performance compared to state-of-the-art gradient-based methods, across a range of tasks. |
|---|