CAREER: Dual Reinforcement Learning: A Unifying Framework with Guarantees

职业：双重强化学习：有保证的统一框架

• 批准号: 2340651
• 负责人: Amy Zhang
• 金额: $ 59.98万
• 依托单位: University of Texas at Austin
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2024
• 资助国家: 美国
• 起止时间: 2024-09-01 至 2029-08-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2340651&HistoricalAwards=false
• 关键词: CAREER; Dual; Reinforcement; Learning; Unifying

Reinforcement learning (RL) holds the promise to automate and improve many real-world processes that require sequential decision-making to optimize some long-term objective, such as self-driving cars, industry automation, recommendation systems, and more recently in natural language processing. There has been much exciting progress in the field of deep reinforcement learning in the past few years, with RL agents demonstrating remarkable performance across a wide range of problem domains. However, to achieve this progress, it is necessary to have access to a fast simulator and tens or hundreds of millions of data points that are collected, trained on, then thrown away. Off-policy methods are an alternative approach, which provide much more data efficiency because they are not restricted to only training on on-policy data and can even be used to train on existing offline data. This suggests that to truly unlock the potential of reinforcement learning, we must develop principled off-policy algorithms. This project is focused on advancing RL by looking at a framework that aims to provide a unified, principled objective that applies to both standard and off-line RL settings and will allow us to efficiently solve large-scale, real-world, sequential decision-making problems.In this project, the PI will examine the dual formulation of this objective, which gives rise to a principled off-policy objective that sidesteps issues present in the more commonly used primal formulation. This objective will lead to algorithms particularly suitable for large state-action spaces, long horizons, and sparse rewards encountered in real-world problems. The PI will explore connections between existing and new imitation learning and reinforcement-learning methods and the proposed framework. The PI will show that both imitation learning and reinforcement learning methods are unified under this objective and present theoretical guarantees for this class of methods. Finally, the PI will extend the dual framework to leverage pre-training and fine tuning for improved sample efficiency. This includes exploring methods for incorporating out-of-domain datasets and multiple modalities in self-supervised pre-training, especially relevant for applications in household robotics.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

强化学习 (RL) 有望实现自动化和改进许多现实世界的流程，这些流程需要顺序决策来优化某些长期目标，例如自动驾驶汽车、行业自动化、推荐系统以及最近的自然语言加工。过去几年，深度强化学习领域取得了许多令人兴奋的进展，强化学习代理在广泛的问题领域中表现出了卓越的性能。然而，要实现这一进步，需要使用快速模拟器以及收集、训练然后丢弃的数千万或数亿个数据点。离策略方法是一种替代方法，它提供了更高的数据效率，因为它们不仅限于对策略数据进行训练，甚至可以用于对现有离线数据进行训练。这表明，要真正释放强化学习的潜力，我们必须开发有原则的离策略算法。该项目的重点是通过研究一个框架来推进强化学习，该框架旨在提供适用于标准和离线强化学习设置的统一、有原则的目标，并使我们能够有效地解决大规模、现实世界的顺序决策问题。在这个项目中，PI 将检查该目标的双重表述，这会产生一个有原则的非政策目标，回避更常用的原始表述中存在的问题。这一目标将导致算法特别适合于现实世界问题中遇到的大状态动作空间、长视野和稀疏奖励。 PI 将探索现有和新的模仿学习和强化学习方法与拟议框架之间的联系。 PI将表明模仿学习和强化学习方法在这一目标下是统一的，并为此类方法提供理论保证。最后，PI 将扩展双重框架，以利用预训练和微调来提高样本效率。这包括探索在自监督预训练中纳入域外数据集和多种模式的方法，特别是与家用机器人应用相关的方法。该奖项反映了 NSF 的法定使命，并通过使用基金会的知识进行评估，被认为值得支持。优点和更广泛的影响审查标准。