Paper Info
Title
Data-efficient Hindsight Off-policy Option Learning
Abstract
We introduce Hindsight Off-policy Options (H02), a data-efficient option learning algorithm. Given any trajectory, HO2 infers likely option choices and backpropagates through the dynamic programming inference procedure to robustly train all policy components off-policy and end-to-end. The approach outperforms existing option learning methods on common benchmarks. To better understand the option framework and disentangle benefits from both temporal and action abstraction, we evaluate ablations with flat policies and mixture policies with comparable optimization. The results highlight the importance of both types of abstraction as well as off-policy training and trust-region constraints, particularly in challenging, simulated 3D robot manipulation tasks from raw pixel inputs. Finally, we intuitively adapt the inference step to investigate the effect of increased temporal abstraction on training with pre-trained options and from scratch.
Year
Venue
DocType
2021
INTERNATIONAL CONFERENCE ON MACHINE LEARNING, VOL 139
Conference
Volume
ISSN
Citations 
139
2640-3498
0
PageRank 
References 
Authors
0.34
0
11
Name
Order
Citations
PageRank
markus wulfmeier1516.86
Dushyant Rao21168.10
Roland Hafner3222.70
Thomas Lampe4212.33
Abbas Abdolmaleki54612.82
Tim Hertweck601.35
M. Neunert7659.95
Dhruva Tirumala800.68
Noah Siegel952.48
Nicolas Heess10176294.77
Martin Riedmiller115655366.29