Collaborative Research: CIF: Medium: MoDL:Toward a Mathematical Foundation of Deep Reinforcement Learning

合作研究：CIF：媒介：MoDL：迈向深度强化学习的数学基础

• 批准号: 2212262
• 负责人: Jason Lee
• 金额: $ 30万
• 依托单位: Princeton University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-10-01 至 2026-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2212262&HistoricalAwards=false
• 关键词: Collaborative; Research; CIF; Medium; MoDL

Deep Reinforcement Learning (DRL), which uses neural networks to solve sequential decision-making problems, has made breakthroughs in real-world applications, such as robotics, gaming, healthcare, and transportation systems. However, current theoretical work on reinforcement learning is restricted to problems with a small number of states; as these results do not cover neural networks, they cannot be used to satisfactorily explain the empirical successes of DRL. This project seeks to bridge this gap by building a mathematical foundation for DRL that leverages ideas from approximation theory, control theory, and optimization theory. This will allow the computational and statistical complexity of DRL to be systematically characterized, and will help with designing more efficient and reliable empirical methods. Education and outreach plans are integrated into this project. Specifically, the investigators will mentor graduate and undergraduate students (some through the STARS program for underrepresented groups at the University of washington), develop new courses and monographs, organize research workshops, and develop course materials for a high school data science and artificial intelligence curriculum. This project has three major components. The first thrust identifies which types of guarantees are achievable by policies for different reinforcement learning problem instances. Concretely, this requires investigating how increasingly structured problem instances enable stronger guarantees for policies; this will be done by using, and further developing, tools from non-convex optimization to describe policies that achieve stationary points, local maxima, and global maxima of the reward function. The second thrust takes the perspective of approximation theory and capacity control to investigate how the neural network complexity can be gradually increased to eventually find the most complex sub-family of neural networks that permit sample-efficient algorithms. The third thrust builds upon the knowledge gained in the first two thrusts, and is devoted to the design of computationally efficient algorithms; this will be done by leveraging tools from optimization theory and by making connections with control theory.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

使用神经网络解决顺序决策问题的深度强化学习（DRL）在现实世界中的应用中取得了突破，例如机器人技术，游戏，医疗保健和运输系统。但是，当前关于强化学习的理论工作仅限于少数州的问题。由于这些结果不涵盖神经网络，因此不能用来令人满意地解释DRL的经验成功。该项目试图通过为DRL建立数学基础来弥合这一差距，该基础利用近似理论，控制理论和优化理论来利用思想。这将使DRL的计算和统计复杂性系统地表征，并有助于设计更有效和可靠的经验方法。教育和外展计划已纳入该项目。具体来说，研究人员将指导毕业生和本科生（有些通过华盛顿大学代表性不足的团体的明星计划），开发新课程和专着，组织研究研讨会，并为高中数据科学和人工智能课程开发课程材料。该项目具有三个主要组成部分。第一个推力可以通过针对不同的强化学习问题实例来确定哪些类型的保证可以实现。具体而言，这需要调查越来越多的结构化问题实例为政策提供更强大的保证；这将通过使用非凸优化的工具以及进一步开发的工具来描述奖励功能的固定点，本地最大值和全球最大值的政策。第二个推力是近似理论和容量控制的观点，以研究如何逐渐增加神经网络的复杂性，以最终找到允许样品效率算法的神经网络中最复杂的亚家族。第三个推力建立在前两个推力中获得的知识，并致力于设计计算有效算法的设计。这将通过利用优化理论的工具和与控制理论建立联系来实现。该奖项反映了NSF的法定任务，并使用基金会的知识分子优点和更广泛的影响标准，被视为值得通过评估来支持。