CAREER: Dual Reinforcement Learning: A Unifying Framework with Guarantees

职业：双重强化学习：有保证的统一框架

• 批准号: 2340651
• 负责人: Amy Zhang
• 金额: $ 59.98万
• 依托单位: University of Texas at Austin
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2024
• 资助国家: 美国
• 起止时间: 2024-09-01 至 2029-08-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2340651&HistoricalAwards=false
• 关键词: CAREER; Dual; Reinforcement; Learning; Unifying

Reinforcement learning (RL) holds the promise to automate and improve many real-world processes that require sequential decision-making to optimize some long-term objective, such as self-driving cars, industry automation, recommendation systems, and more recently in natural language processing. There has been much exciting progress in the field of deep reinforcement learning in the past few years, with RL agents demonstrating remarkable performance across a wide range of problem domains. However, to achieve this progress, it is necessary to have access to a fast simulator and tens or hundreds of millions of data points that are collected, trained on, then thrown away. Off-policy methods are an alternative approach, which provide much more data efficiency because they are not restricted to only training on on-policy data and can even be used to train on existing offline data. This suggests that to truly unlock the potential of reinforcement learning, we must develop principled off-policy algorithms. This project is focused on advancing RL by looking at a framework that aims to provide a unified, principled objective that applies to both standard and off-line RL settings and will allow us to efficiently solve large-scale, real-world, sequential decision-making problems.In this project, the PI will examine the dual formulation of this objective, which gives rise to a principled off-policy objective that sidesteps issues present in the more commonly used primal formulation. This objective will lead to algorithms particularly suitable for large state-action spaces, long horizons, and sparse rewards encountered in real-world problems. The PI will explore connections between existing and new imitation learning and reinforcement-learning methods and the proposed framework. The PI will show that both imitation learning and reinforcement learning methods are unified under this objective and present theoretical guarantees for this class of methods. Finally, the PI will extend the dual framework to leverage pre-training and fine tuning for improved sample efficiency. This includes exploring methods for incorporating out-of-domain datasets and multiple modalities in self-supervised pre-training, especially relevant for applications in household robotics.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

强化学习（RL）有望自动化和改善许多现实世界的过程，这些过程需要顺序决策以优化一些长期目标，例如自动驾驶汽车，行业自动化，推荐系统以及最近的自然语言处理。在过去的几年中，深入的增强学习领域取得了令人兴奋的进步，RL代理在广泛的问题领域表现出了出色的表现。但是，为了实现这一进展，有必要访问快速模拟器，以及收集，训练，然后扔掉的数千万或数亿个数据点。政体方法是一种替代方法，它提供了更高的数据效率，因为它们不仅限于对政策数据的培训，甚至可以用于培训现有的离线数据。这表明，要真正释放强化学习的潜力，我们必须开发有原则的非政策算法。该项目的重点是通过查看一个旨在提供统一的，原则性目标的框架来推进RL，该框架适用于标准和离线RL设置，并将使我们能够有效地解决大型，真实的，顺序的决策问题。在此项目中，PI将对这个目标的双重形式提出了一个普通的构造，以实现较大的目标，以确定较大的目标。该目标将导致算法特别适用于在现实世界中遇到的大型州行动空间，较长的视野和稀疏的奖励。 PI将探索现有和新的模仿学习与加强学习方法与拟议框架之间的联系。 PI将表明，模仿学习和强化学习方法都是在此目标下统一的，并为这类方法提供了理论保证。最后，PI将扩展双重框架以利用预训练和微调以提高样品效率。这包括探索用于合并域外数据集的方法以及在自我监管的预培训中的多种模式，尤其与家庭机器人技术中的应用有关。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的知识分子优点和更广泛的审查标准通过评估来通过评估来支持的。