Collaborative Research: CIF: Medium: MoDL:Toward a Mathematical Foundation of Deep Reinforcement Learning

合作研究：CIF：媒介：MoDL：迈向深度强化学习的数学基础

• 批准号: 2212262
• 负责人: Jason Lee
• 金额: $ 30万
• 依托单位: Princeton University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-10-01 至 2026-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2212262&HistoricalAwards=false
• 关键词: Collaborative; Research; CIF; Medium; MoDL

Deep Reinforcement Learning (DRL), which uses neural networks to solve sequential decision-making problems, has made breakthroughs in real-world applications, such as robotics, gaming, healthcare, and transportation systems. However, current theoretical work on reinforcement learning is restricted to problems with a small number of states; as these results do not cover neural networks, they cannot be used to satisfactorily explain the empirical successes of DRL. This project seeks to bridge this gap by building a mathematical foundation for DRL that leverages ideas from approximation theory, control theory, and optimization theory. This will allow the computational and statistical complexity of DRL to be systematically characterized, and will help with designing more efficient and reliable empirical methods. Education and outreach plans are integrated into this project. Specifically, the investigators will mentor graduate and undergraduate students (some through the STARS program for underrepresented groups at the University of washington), develop new courses and monographs, organize research workshops, and develop course materials for a high school data science and artificial intelligence curriculum. This project has three major components. The first thrust identifies which types of guarantees are achievable by policies for different reinforcement learning problem instances. Concretely, this requires investigating how increasingly structured problem instances enable stronger guarantees for policies; this will be done by using, and further developing, tools from non-convex optimization to describe policies that achieve stationary points, local maxima, and global maxima of the reward function. The second thrust takes the perspective of approximation theory and capacity control to investigate how the neural network complexity can be gradually increased to eventually find the most complex sub-family of neural networks that permit sample-efficient algorithms. The third thrust builds upon the knowledge gained in the first two thrusts, and is devoted to the design of computationally efficient algorithms; this will be done by leveraging tools from optimization theory and by making connections with control theory.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度强化学习 (DRL) 使用神经网络解决顺序决策问题，在机器人、游戏、医疗保健和交通系统等现实应用中取得了突破。然而，当前强化学习的理论工作仅限于少数状态的问题；由于这些结果不涵盖神经网络，因此它们不能用来令人满意地解释 DRL 的经验成功。该项目旨在利用近似理论、控制理论和优化理论的思想，为 DRL 建立数学基础，从而弥补这一差距。这将使 DRL 的计算和统计复杂性得到系统表征，并有助于设计更高效、更可靠的经验方法。教育和推广计划已纳入该项目。具体来说，研究人员将指导研究生和本科生（其中一些是通过华盛顿大学针对代表性不足群体的 STARS 计划）、开发新课程和专着、组织研究研讨会以及为高中数据科学和人工智能课程开发课程材料。该项目由三个主要部分组成。第一个主旨确定了针对不同强化学习问题实例的策略可以实现哪些类型的保证。具体来说，这需要研究日益结构化的问题实例如何为政策提供更有力的保障；这将通过使用并进一步开发非凸优化工具来描述实现奖励函数的固定点、局部最大值和全局最大值的策略来完成。第二个主旨从近似理论和容量控制的角度来研究如何逐渐增加神经网络的复杂性，以最终找到允许样本高效算法的最复杂的神经网络子族。第三个主旨建立在前两个主旨所获得的知识的基础上，致力于计算高效的算法的设计；这将通过利用优化理论的工具并与控制理论建立联系来完成。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。