Reinforcement Learning for Finite Horizons (ReLeaF)

有限视野强化学习 (ReLeaF)

基本信息

批准号：
EP/X021513/1
负责人：
Sven Schewe
金额：
$ 26万
依托单位：
University of Liverpool
依托单位国家：
英国
项目类别：
Fellowship
财政年份：
2022
资助国家：
英国
起止时间：
2022 至无数据
项目状态：
未结题

来源：
https://gtr.ukri.org/projects?ref=EP%2FX021513%2F1
关键词：
Reinforcement Learning Finite Horizons ReLeaF

项目摘要

Reinforcement learning (RL) is a technique for learning how to take actions in an initially unknown environment in order to optimise an expected outcome, which is modelled through the notion of maximising an accumulative reward. Learning algorithms with goals written as temporal specifications have three key ingredients: the translation from the specification to appropriate finite automata; the translation of these finite automata to reward structures, such that a strategy that provides optimal rewards is guaranteed to provide optimal control; and a wrapper into a discounting scheme that, for appropriate parameters, will ensure that a learner converge to an optimal strategy.We will consider the RL problems for a popular specification language used in automation and motion planning, the finite horizon linear time temporal logic LTLf. In particular, we will study model-free RL algorithms, which are more suitable to real-world applications where the behaviour of the environment is hard to predict, than its model-based counterpart. We will propose learning algorithms that provide translations from finite horizon LTL to reward structures with formal guarantees of satisfying the given goals for environments modelled as Markov Decision Processes (MDPs). We will extend our techniques to infinite-state MDPs, including variations where formal guarantees can be provided -- like countable, finitely branching MDPs -- and study conditions for our techniques to provide guarantees in more general classes, such as smoothness guarantees for compact MDPs. We will complement these lines of research by looking at goals with constraints. This is effectively considering prioritised goals, where meeting safety constraints takes precedence, while other properties -- such as efficiency -- are considered as tie-breakers among strategies that provide the same safety guarantees.

强化学习 (RL) 是一种学习如何在最初未知的环境中采取行动以优化预期结果的技术，它通过最大化累积奖励的概念进行建模。将目标写为时间规范的学习算法具有三个关键要素：从规范到适当的有限自动机的转换；将这些有限自动机转换为奖励结构，从而保证提供最佳奖励的策略能够提供最佳控制；以及折扣方案的包装器，对于适当的参数，该方案将确保学习器收敛到最佳策略。我们将考虑自动化和运动规划中使用的流行规范语言（有限水平线性时间时序逻辑 LTLf）的 RL 问题。特别是，我们将研究无模型强化学习算法，与基于模型的对应算法相比，该算法更适合环境行为难以预测的现实世界应用。我们将提出学习算法，提供从有限水平 LTL 到奖励结构的转换，并正式保证满足建模为马尔可夫决策过程 (MDP) 的环境的给定目标。我们将把我们的技术扩展到无限状态 MDP，包括可以提供形式保证的变体（例如可数、有限分支 MDP），并研究我们的技术在更一般类别中提供保证的条件，例如紧凑 MDP 的平滑保证。我们将通过考虑有限制的目标来补充这些研究方向。这是有效地考虑优先目标，其中满足安全约束优先，而其他属性（例如效率）被视为提供相同安全保证的策略之间的决定性因素。