CAREER: Towards Real-world Reinforcement Learning

职业：走向现实世界的强化学习

• 批准号: 2339395
• 负责人: Wen Sun
• 金额: $ 60万
• 依托单位: Cornell University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2024
• 资助国家: 美国
• 起止时间: 2024-03-01 至 2029-02-28
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2339395&HistoricalAwards=false
• 关键词: CAREER; Towards; Real; world; Reinforcement

Reinforcement learning (RL) is one of the most important paradigms for modeling data-driven decision-making. Recent years have witnessed several empirical successes of RL, such as RL agents that outperform humans in video and board games. However, many empirical RL algorithms today often require many training examples to learn and can produce unreliable solutions (solutions that exhibit catastrophic failures, for example). While these issues are typically not problematic when training RL agents in simulators, they pose significant difficulties when deploying RL to real-world problems where data (including human feedback) is expensive, and reliability is essential. The main novelty of this project will be the development of new RL algorithms that can learn efficiently (from as few training data points as possible) and reliably (avoid catastrophic failures with high probability). The development of such RL algorithms can expand the applications of RL systems from simulation to real-world applications where data is expensive to collect and safety is critical. In autonomous driving, the developed technologies can make self-driving cars adapt to new road conditions safely by making fewer mistakes. In generative Artificial Intelligence (AI), efficient and reliable RL algorithms that can learn from rich human feedback will enable better human-AI alignment, making AI systems improve reliably and safely under human guidance.The main research goal of this project is to enable real-world RL by advancing RL techniques, theoretically and empirically. The critical innovation in the project is to develop safe and efficient RL algorithms by leveraging specific problem structures and rich human feedback. The project has three main thrusts. First, the project will establish risk-averse RL algorithms that are provably correct and scalable to high dimensional data. Second, the project will develop RL algorithms that can leverage common problem-specific structures for improved sample efficiency. Third, the project will create new algorithms for RL with rich feedback beyond scalar rewards (including preference-based feedback and positive demonstrations). In addition to the proposed work on algorithmic advancements, the project will focus on their deployment to real-world problems, including database query optimization and optimizing generative models such as Large Language Models and Diffusion Models.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

强化学习（RL）是建模数据驱动决策的最重要范例之一。近年来，RL取得了几项经验成功，例如在视频和棋盘游戏中超过人类的RL代理。但是，如今许多经验RL算法通常需要许多培训示例来学习并可以产生不可靠的解决方案（例如，显示出灾难性故障的解决方案）。尽管在模拟器中训练RL代理时，这些问题通常不会有问题，但在将RL部署到现实世界中的问题时，它们在数据（包括人反馈）很昂贵且可靠性至关重要的情况下遇到了很大的困难。该项目的主要新颖性将是开发新的RL算法，这些算法可以有效地学习（从尽可能少的训练数据点）学习并可靠地学习（避免使用高概率的灾难性失败）。此类RL算法的开发可以扩展RL系统的应用程序从模拟到现实世界应用程序，在这些应用程序中，数据昂贵，安全至关重要。在自动驾驶中，开发的技术可以使自动驾驶汽车通过犯下更少的错误来安全地适应新的道路条件。在生成人工智能（AI）中，可以从丰富的人类反馈中学习的有效且可靠的RL算法将使人类AI系统的一致性更好，从而使AI系统在人类的指导下可靠，安全地改善。该项目的主要研究目标是通过在理论上和经验上促进RL技术来启用现实世界中的RL。该项目的关键创新是通过利用特定的问题结构和丰富的人类反馈来开发安全有效的RL算法。该项目有三个主要推力。首先，该项目将建立规避风险的RL算法，这些算法可证明是正确且可扩展到高维数据的。其次，该项目将开发RL算法，这些算法可以利用特定于问题的结构来提高样品效率。第三，该项目将为RL创建新的算法，并具有超越标量奖励的丰富反馈（包括基于偏好的反馈和积极的演示）。除了提出有关算法进步的拟议工作外，该项目还将集中在其部署到现实世界中的问题上，包括数据库查询优化和优化生成模型，例如大语言模型和扩散模型。该奖项反映了NSF的法定任务，并被认为是通过基金会的知识优点和广泛的cribitia进行评估，以评估值得通过评估来进行评估。