Publications
Publications are listed in reversed chronological order.
2023
- Hyperparameters in Reinforcement Learning and How To Tune ThemTheresa Eimer, Marius Lindauer, and Roberta RaileanuProceedings of the Fortieth International Conference on Machine Learning, Jul 2023
In order to improve reproducibility, deep reinforcement learning (RL) has been adopting better scientific practices such as standardized evaluation metrics and reporting. However, the process of hyperparameter optimization still varies widely across papers, which makes it challenging to compare RL algorithms fairly. In this paper, we show that hyperparameter choices in RL can significantly affect the agent’s final performance and sample efficiency, and that the hyperparameter landscape can strongly depend on the tuning seed which may lead to overfitting. We therefore propose adopting established best practices from AutoML, such as the separation of tuning and testing seeds, as well as principled hyperparameter optimization (HPO) across a broad search space. We support this by comparing multiple state-of-the-art HPO tools on a range of RL algorithms and environments to their hand-tuned counterparts, demonstrating that HPO approaches often have higher performance and lower compute overhead. As a result of our findings, we recommend a set of best practices for the RL community, which should result in stronger empirical results with fewer computational costs, better reproducibility, and thus faster progress. In order to encourage the adoption of these practices, we provide plug-and-play implementations of the tuning algorithms used in this paper.
@article{eimer-icml23, author = {Eimer, Theresa and Lindauer, Marius and Raileanu, Roberta}, title = {Hyperparameters in Reinforcement Learning and How To Tune Them}, journal = {Proceedings of the Fortieth International Conference on Machine Learning}, month = jul, year = {2023}, } - Contextualize Me - The Case for Context in Reinforcement LearningCarolin Benjamins, Theresa Eimer, Frederik Schubert, Sebastian Döhler, Aditya Mohan, André Biedenkapp, Bodo Rosenhahn, Frank Hutter, and Marius LindauerTransactions on Machine Learning Research, Jul 2023
While Reinforcement Learning (RL) has made great strides towards solving increasingly complicated problems, many algorithms are still brittle to even slight environmental changes. Contextual Reinforcement Learning (cRL) provides a framework to model such changes in a principled manner, thereby enabling flexible, precise and interpretable task specification and generation. Our goal is to show how the framework of cRL contributes to improving zero-shot generalization in RL through meaningful benchmarks and structured reasoning about generalization tasks. We confirm the insight that optimal behavior in cRL requires context information, as in other related areas of partial observability. To empirically validate this in the cRL framework, we provide various context-extended versions of common RL environments. They are part of the first benchmark library, CARL, designed for generalization based on cRL extensions of popular benchmarks, which we propose as a testbed to further study general agents. We show that in the contextual setting, even simple RL environments become challenging - and that naive solutions are not enough to generalize across complex context spaces.
@article{benjamins-tmlr23, author = {Benjamins, Carolin and Eimer, Theresa and Schubert, Frederik and Döhler, Sebastian and Mohan, Aditya and Biedenkapp, André and Rosenhahn, Bodo and Hutter, Frank and Lindauer, Marius}, title = {Contextualize Me - The Case for Context in Reinforcement Learning}, journal = {Transactions on Machine Learning Research}, year = {2023}, }
2022
- Automated Reinforcement Learning (AutoRL): A Survey and Open ProblemsJack Parker-Holder, Raghu Rajan, Xingyou Song, André Biedenkapp, Yingjie Miao, Theresa Eimer, Baohe Zhang, Vu Nguyen, Roberto Calandra, Aleksandra Faust, Frank Hutter, and Marius LindauerJournal of Artificial Intelligence Research (JAIR), Jul 2022
The combination of Reinforcement Learning (RL) with deep learning has led to a series of impressive feats, with many believing (deep) RL provides a path towards generally capable agents. However, the success of RL agents is often highly sensitive to design choices in the training process, which may require tedious and error-prone manual tuning. This makes it challenging to use RL for new problems, while also limits its full potential. In many other areas of machine learning, AutoML has shown it is possible to automate such design choices and has also yielded promising initial results when applied to RL. However, Automated Reinforcement Learning (AutoRL) involves not only standard applications of AutoML but also includes additional challenges unique to RL, that naturally produce a different set of methods. As such, AutoRL has been emerging as an important area of research in RL, providing promise in a variety of applications from RNA design to playing games such as Go. Given the diversity of methods and environments considered in RL, much of the research has been conducted in distinct subfields, ranging from meta-learning to evolution. In this survey we seek to unify the field of AutoRL, we provide a common taxonomy, discuss each area in detail and pose open problems which would be of interest to researchers going forward.
@article{parker-holder-jair22, author = {Parker{-}Holder, Jack and Rajan, Raghu and Song, Xingyou and Biedenkapp, André and Miao, Yingjie and Eimer, Theresa and Zhang, Baohe and Nguyen, Vu and Calandra, Roberto and Faust, Aleksandra and Hutter, Frank and Lindauer, Marius}, title = {Automated Reinforcement Learning (AutoRL): {A} Survey and Open Problems}, year = {2022}, journal = {Journal of Artificial Intelligence Research (JAIR)}, pages = {517-568}, volume = {74} } - Automated Dynamic Algorithm ConfigurationSteven Adriaensen, André Biedenkapp, Gresa Shala, Noor Awad, Theresa Eimer, Marius Lindauer, and Frank HutterJournal of Artificial Intelligence Research, Jul 2022
The performance of an algorithm often critically depends on its parameter configuration. While a variety of automated algorithm configuration methods have been proposed to relieve users from the tedious and error-prone task of manually tuning parameters, there is still a lot of untapped potential as the learned configuration is static, i.e., parameter settings remain fixed throughout the run. However, it has been shown that some algorithm parameters are best adjusted dynamically during execution, e.g., to adapt to the current part of the optimization landscape. Thus far, this is most commonly achieved through hand-crafted heuristics. A promising recent alternative is to automatically learn such dynamic parameter adaptation policies from data. In this article, we give the first comprehensive account of this new field of automated dynamic algorithm configuration (DAC), present a series of recent advances, and provide a solid foundation for future research in this field. Specifically, we (i) situate DAC in the broader historical context of AI research; (ii) formalize DAC as a computational problem; (iii) identify the methods used in prior-art to tackle this problem; (iv) conduct empirical case studies for using DAC in evolutionary optimization, AI planning, and machine learning.
@article{adriaensen-jair22, author = {Adriaensen, Steven and Biedenkapp, André and Shala, Gresa and Awad, Noor and Eimer, Theresa and Lindauer, Marius and Hutter, Frank}, title = {Automated Dynamic Algorithm Configuration}, journal = {Journal of Artificial Intelligence Research}, volume = {75}, pages = {1633--1699}, year = {2022}, }
2021
- DACBench: A Benchmark Library for Dynamic Algorithm ConfigurationTheresa Eimer, André Biedenkapp, Maximilian Reimer, Steven Adriaensen, Frank Hutter, and Marius LindauerIn Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI’21), Aug 2021
Dynamic Algorithm Configuration (DAC) aims to dynamically control a target algorithm’s hyperparameters in order to improve its performance. Several theoretical and empirical results have demonstrated the benefits of dynamically controlling hyperparameters in domains like evolutionary computation, AI Planning or deep learning. Replicating these results, as well as studying new methods for DAC, however, is difficult since existing benchmarks are often specialized and incompatible with the same interfaces. To facilitate benchmarking and thus research on DAC, we propose DACBench, a benchmark library that seeks to collect and standardize existing DAC benchmarks from different AI domains, as well as provide a template for new ones. For the design of DACBench, we focused on important desiderata, such as (i) flexibility, (ii) reproducibility, (iii) extensibility and (iv) automatic documentation and visualization. To show the potential, broad applicability and challenges of DAC, we explore how a set of six initial benchmarks compare in several dimensions of difficulty.
@inproceedings{eimer-ijcai21, author = {Eimer, Theresa and Biedenkapp, André and Reimer, Maximilian and Adriaensen, Steven and Hutter, Frank and Lindauer, Marius}, title = {DACBench: A Benchmark Library for Dynamic Algorithm Configuration}, booktitle = {Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence ({IJCAI}'21)}, year = {2021}, month = aug, publisher = {ijcai.org}, talk = {https://www.youtube.com/watch?v=-G-hLmBI4WM} } - Self-Paced Context Evaluation for Contextual Reinforcement LearningIn Proceedings of the Thirty-eighth International Conference on Machine Learning, Jul 2021
Reinforcement learning (RL) has made a lot of advances for solving a single problem in a given environment; but learning policies that generalize to unseen variations of a problem remains challenging. To improve sample efficiency for learning on such instances of a problem domain, we present Self-Paced Context Evaluation (SPaCE). Based on self-paced learning, \spc automatically generates \task curricula online with little computational overhead. To this end, SPaCE leverages information contained in state values during training to accelerate and improve training performance as well as generalization capabilities to new instances from the same problem domain. Nevertheless, SPaCE is independent of the problem domain at hand and can be applied on top of any RL agent with state-value function approximation. We demonstrate SPaCE’s ability to speed up learning of different value-based RL agents on two environments, showing better generalization capabilities and up to 10x faster learning compared to naive approaches such as round robin or SPDRL, as the closest state-of-the-art approach.
@inproceedings{eimer-icml21, author = {Eimer, Theresa and Biedenkapp, André and Hutter, Frank and Lindauer, Marius}, title = {Self-Paced Context Evaluation for Contextual Reinforcement Learning}, booktitle = {Proceedings of the Thirty-eighth International Conference on Machine Learning}, year = {2021}, month = jul, } - Automatic Risk Adaption in Distributional Reinforcement LearningIn Workshop on Reinforcement Learning for Real Life (RL4RealLife@ICML’21), Jul 2021
The use of Reinforcement Learning (RL) agents in practical applications requires the consideration of suboptimal outcomes, depending on the familiarity of the agent with its environment. This is especially important in safety-critical environments, where errors can lead to high costs or damage. In distributional RL, the risk-sensitivity can be controlled via different distortion measures of the estimated return distribution. However, these distortion functions require an estimate of the risk level, which is difficult to obtain and depends on the current state. In this work, we demonstrate the suboptimality of a static risk level estimation and propose a method to dynamically select risk levels at each environment step. Our method ARA (Automatic Risk Adaptation) estimates the appropriate risk level in both known and unknown environments using a Random Network Distillation error. We show reduced failure rates by up to a factor of 7 and improved generalization performance by up to 14 percent compared to both risk-aware and risk-agnostic agents in several locomotion environments.
@inproceedings{schubert-rl4rlicml21, author = {Schubert, Frederik and Eimer, Theresa and Rosenhahn, Bodo and Lindauer, Marius}, title = {Automatic Risk Adaption in Distributional Reinforcement Learning}, booktitle = {Workshop on Reinforcement Learning for Real Life ({RL4RealLife@ICML}'21)}, year = {2021}, month = jul, } - Hyperparameters in Contextual RL are Highly SituationalTheresa Eimer, Carolin Benjamins, and Marius LindauerIn Ecological Theory of RL Workshop NeurIPS, Dez 2021
Although Reinforcement Learning (RL) has shown impressive results in games and simulation, real-world application of RL suffers from its instability under changing environment conditions and hyperparameters. We give a first impression of the extent of this instability by showing that the hyperparameters found by automatic hyperparameter optimization (HPO) methods are not only dependent on the problem at hand, but even on how well the state describes the environment dynamics. Specifically, we show that agents in contextual RL require different hyperparameters if they are shown how environmental factors change. In addition, finding adequate hyperparameter configurations is not equally easy for both settings, further highlighting the need for research into how hyperparameters influence learning and generalization in RL.
@inproceedings{eimer-ecorl21, author = {Eimer, Theresa and Benjamins, Carolin and Lindauer, Marius}, title = {Hyperparameters in Contextual RL are Highly Situational}, booktitle = {Ecological Theory of RL Workshop NeurIPS}, year = {2021}, month = dez, }
2020
- Dynamic Algorithm Configuration: Foundation of a New Meta-Algorithmic FrameworkIn Proceedings of the European Conference on Artificial Intelligence (ECAI), Jun 2020
The performance of many algorithms in the fields of hard combinatorial problem solving, machine learning or AI in general depends on parameter tuning. Automated methods have been proposed to alleviate users from the tedious and error-prone task of manually searching for performance-optimized configurations across a set of problem instances. However, there is still a lot of untapped potential through adjusting an algorithm’s parameters online since different parameter values can be optimal at different stages of the algorithm. Prior work showed that reinforcement learning is an effective approach to learn policies for online adjustments of algorithm parameters in a data-driven way. We extend that approach by formulating the resulting dynamic algorithm configuration as a contextual MDP, such that RL not only learns a policy for a single instance, but across a set of instances. To lay the foundation for studying dynamic algorithm configuration with RL in a controlled setting, we propose white-box benchmarks covering major aspects that make dynamic algorithm configuration a hard problem in practice and study the per- formance of various types of configuration strategies for them. On these white-box benchmarks, we show that (i) RL is a robust candidate for learning configuration policies, outperforming standard pa- rameter optimization approaches, such as classical algorithm configuration; (ii) based on function approximation, RL agents can learn to generalize to new types of instances; and (iii) self-paced learning can substantially improve the performance by selecting a useful sequence of training instances automatically.
@inproceedings{biedenkapp-ecai20, author = {Biedenkapp, André and Bozkurt, H. Furkan and Eimer, Theresa and Hutter, Frank and Lindauer, Marius}, title = {Dynamic {A}lgorithm {C}onfiguration: {F}oundation of a {N}ew {M}eta-{A}lgorithmic {F}ramework}, booktitle = {Proceedings of the European Conference on Artificial Intelligence (ECAI)}, year = {2020}, month = jun, talk = {https://www.youtube.com/watch?v=wxPYtSGT05s&feature=youtu.be}, }