CAREER: Reinforcement Learning for Recursive Markov Decision Processes and Beyond

职业：递归马尔可夫决策过程及其他的强化学习

• 批准号: 2146563
• 负责人: Ashutosh Trivedi
• 金额: $ 59.66万
• 依托单位: University of Colorado at Boulder
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-05-01 至 2027-04-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2146563&HistoricalAwards=false
• 关键词: CAREER; Reinforcement; Learning; Recursive; Markov

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).Reinforcement Learning (RL) is a sampling-based approach to optimization of Markov decision processes (MDPs), where agents rely on rewards to discover optimal solutions. When combined with powerful approximation schemes (e.g., deep neural networks), RL has been effective in highly complex tasks traditionally considered beyond reach of Artificial Intelligence. However, its sensitivity to the approximation parameters makes RL difficult to use (significant Machine Learning expertise is demanded of the programmer) and difficult to trust (manual approximations can invalidate guarantees). The vision of this project is to democratize RL by developing principled methodologies and powerful tools to improve the usability and trustworthiness of RL-based programming at scale. These research objectives are complemented by efforts to integrate the foundations of RL-based computability in CS education and to explore the role of RL-based programming in CS education.Approximation in RL is needed because RL algorithms with guaranteed convergence work on finite MDPs, and yet scale poorly. Approximation affects usability and trustworthiness. This proposal identifies two goals addressing both concerns: 1) to discover convergent RL beyond finite MDPs and 2) to develop abstraction-based approaches for RL with rigorous optimization guarantees. The success of the proposed approaches will be evaluated by their ability to handle systems at scale. The algorithms and datasets will be disseminated as open-source software. The proposed research makes fundamental contributions to three disciplines: formal methods, machine learning, and control theory; at the same time, it takes fundamental, concrete steps towards broadening participation in computing by making RL-based programming easier and more inclusive.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该奖项的全部或部分资金来源于《2021 年美国救援计划法案》（公法 117-2）。强化学习 (RL) 是一种基于采样的马尔可夫决策过程 (MDP) 优化方法，其中智能体依赖于奖励发现最佳解决方案。当与强大的近似方案（例如深度神经网络）相结合时，强化学习在传统上被认为超出人工智能能力范围的高度复杂任务中非常有效。然而，它对逼近参数的敏感性使得强化学习难以使用（需要程序员具备丰富的机器学习专业知识）并且难以信任（手动逼近可能会使保证失效）。该项目的愿景是通过开发有原则的方法和强大的工具来大规模提高基于强化学习的编程的可用性和可信度，从而使强化学习民主化。这些研究目标还通过努力将基于 RL 的可计算性基础整合到 CS 教育中，并探索基于 RL 的编程在 CS 教育中的作用来补充。需要 RL 逼近，因为保证收敛的 RL 算法适用于有限 MDP，并且但规模很差。近似值会影响可用性和可信度。该提案确定了解决这两个问题的两个目标：1）发现超越有限 MDP 的收敛强化学习；2）开发具有严格优化保证的基于抽象的强化学习方法。所提出的方法是否成功将通过其大规模处理系统的能力来评估。算法和数据集将作为开源软件传播。 拟议的研究对三个学科做出了基础性贡献：形式方法、机器学习和控制理论；与此同时，它采取了根本性的、具体的步骤，通过使基于强化学习的编程变得更容易、更具包容性，来扩大对计算的参与。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力优势和更广泛的影响进行评估，被认为值得支持。审查标准。

项目成果

Reinforcement Learning with Guarantees that Hold for Ever

强化学习，保证永远有效

• 发表时间: 2022-09
• 期刊: Formal Methods for Industrial Critical Systems. FMICS 2022
• 作者: Hahn, Ernst Moritz;Perez, Mateo;Schewe, Sven;Somenzi, Fabio;Trivedi, Ashutosh;Wojtczak, Dominik
• 通讯作者: Wojtczak, Dominik

The Octatope Abstract Domain for Verification of Neural Networks.

用于验证神经网络的八位位抽象域。

• 发表时间: 2023-03
• 期刊: Formal Methods. FM 2023.
• 作者: Bak, S.;Dohmen, T.;Subramani, K.;Trivedi, A.;Velasquez, A.;Wojciechowski, P.
• 通讯作者: Wojciechowski, P.

Recursive Reinforcement Learning

递归强化学习

• 发表时间: 2022-11
• 期刊: Conference on Neural Information Processing Systems (NeurIPS 2022
• 作者: Hahn, Ernst Moritz;Perez, Mateo;Schewe, Sven;Somenzi, Fabio;Trivedi, Ashutosh;Wojtczak, Dominik
• 通讯作者: Wojtczak, Dominik

Optimal Repair for Omega-Regular Properties

Omega-Regular 特性的最佳修复

• 发表时间: 2022-10
• 期刊: Automated Technology for Verification and Analysis. ATVA 2022.
• 作者: Dave, V.;Krishna, S.;Murali, V.;Trivedi, A.
• 通讯作者: Trivedi, A.

An Impossibility Result in Automata-Theoretic Reinforcement Learning.

自动机理论强化学习的不可能结果。

• 发表时间: 2022-10
• 期刊: International Symposium on Automated Technology for Verification and Analysis (ATVA 2022
• 作者: Hahn, Ernst Moritz;Perez, Mateo;Schewe, Sven;Somenzi, Fabio;Trivedi, Ashutosh;Wojtczak, Dominik
• 通讯作者: Wojtczak, Dominik