Collaborative Research: SLES: Safe Distributional-Reinforcement Learning-Enabled Systems: Theories, Algorithms, and Experiments

协作研究：SLES：安全的分布式强化学习系统：理论、算法和实验

• 批准号: 2331782
• 负责人: Xian Yu
• 金额: $ 37.5万
• 依托单位: Ohio State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-10-01 至 2027-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2331782&HistoricalAwards=false
• 关键词: Collaborative; Research; SLES; Safe; Distributional

Reinforcement learning (RL), with its success in automation and robotics, has been widely viewed as one of the most important technologies for next-generation, learning-enabled systems. For example, 6G networking systems, autonomous driving, digital healthcare, and smart cities are all enabled by RL. However, despite the significant advances over the last few decades, a major obstacle in applying RL in practice is the lack of “safety'' guarantees such as robustness, resilience to tail-risks, operational constraints, etc. This is because the traditional RL only aims at maximizing cumulative reward. While it is possible to add penalties to rewards in a traditional RL algorithm to discourage unsafe actions, many safety constraints, such as chance constraints, cannot be simply treated as penalties. This project develops foundational technologies for safe RL-enabled systems based on Distributional Reinforcement Learning (DRL), which learns the optimal policy. While developing the foundation of DRL for safe learning-enabled systems, research and education are integrated by including new theories and algorithms developed in this project into their graduate-level courses. All team members have been regularly supervising undergraduate students and students from underrepresented groups. The team continues to leverage Women's Place at Ohio State University and the Women in Science and Engineering Program at Arizona State University to enhance the broader participation of women students and researchers. This project focuses on a comprehensive approach for the end-to-end safety of DRL-enabled systems. The end-to-end safety includes (i) policy safety: learn a safe policy to avoid the occurrence of catastrophic outcomes (corresponds to risk-sensitive RL); (ii) exploration safety -- learn a safe policy safely by avoiding dangerous actions during exploration/learning (corresponds to online RL); and (iii) environmental safety -- learn a policy that is robust to parametric uncertainty (environment change). This project includes four thrusts. Thrust 1 (Foundation of constrained DRL) aims to establish theoretical foundations of risk sensitive constrained DRL and focuses on policy and environmental safety. Thrust 2 (Online constrained DRL) considers safe online learning and decision-making and focuses on exploration safety and environmental safety when learning a safe DRL policy. Thrust 3 (Physics-Enhanced constrained DRL) exploits physics to enhance end-to-end safety. These three thrusts on foundational research are interdependent, but each focuses on a unique aspect of safe RL-enabled systems and addresses multiple safety notions. The fourth thrust will provide comprehensive validation with both high-fidelity simulations and real-world experiments using unmanned aerial vehicles.This research is supported by a partnership between the National Science Foundation and Open Philanthropy.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

强化学习（RL）凭借其在自动化和机器人技术方面的成功，已被广泛视为下一代，学习的系统最重要的技术之一。例如，RL启用了6G网络系统，自动驾驶，数字医疗保健和智能城市。然而，尽管在过去几十年中取得了重大进展，但在实践中应用RL的主要障碍是缺乏“安全性”的保证，例如稳健性，对尾巴风险的韧性，操作限制等。这是因为传统RL仅旨在最大程度地提高累积奖励。虽然可能会增加限制的限制。限制不能简单地视为基于分配加强学习的安全系统的基础技术（DRL），在开发最佳的政策的同时，为安全学习的系统开发了DRL的基础，研究和教育是由纽约的整个项目组成的。代表性不足的团队继续利用俄亥俄州立大学的女性地位，亚利桑那州立大学的科学与工程计划中的妇女地位来增强女学生和研究人员的广泛参与。该项目重点介绍了支持DRL系统的端到端安全性的全面方法。端到端安全包括（i）政策安全：学习安全政策以避免发生灾难性结果（对应于风险敏感的RL）； （ii）勘探安全 - 通过在探索/学习过程中避免危险行动（对应在线RL），可以安全地学习安全政策； （iii）环境安全 - 学习一项针对参数不确定性（环境变化）的政策。该项目包括四个推力。推力1（受约束DRL的基础）旨在建立对风险敏感受限的DRL的理论基础，并专注于政策和环境安全。推力2（在线约束DRL）考虑安全的在线学习和决策，并专注于学习安全的DRL政策时的勘探安全和环境安全。推力3（物理增强的约束DRL）利用物理来增强端到端的安全性。基础研究的这三个推力是相互依存的，但每个推力都集中在安全RL的系统的独特方面，并解决了多个安全说明。 The fourth thrust will provide comprehensive validation with both high-fidelity simulations and real-world experiments using unmanned aerial vehicles.This research is supported by a partnership between the National Science Foundation and Open Philanthropy.This award reflects NSF's statutory mission and has been deemed honestly of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.