CPS: Medium: Collaborative Research: Provably Safe and Robust Multi-Agent Reinforcement Learning with Applications in Urban Air Mobility

CPS：中：协作研究：可证明安全且鲁棒的多智能体强化学习及其在城市空中交通中的应用

• 批准号: 2312092
• 负责人: Peng Wei
• 金额: $ 40万
• 依托单位: George Washington University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-06-01 至 2026-05-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2312092&HistoricalAwards=false
• 关键词: CPS; Medium; Collaborative; Research; Provably

This Cyber-Physical Systems (CPS) project aims at designing theories and algorithms for scalable multi-agent planning and control to support safety-critical autonomous eVTOL aircraft in high-throughput, uncertain and dynamic environments. Urban Air Mobility (UAM) is an emerging air transportation mode in which electrical vertical take-off and landing (eVTOL) aircraft will safely and efficiently transport passengers and cargo within urban areas. Guidance from the White House, the National Academy of Engineering, and the US Congress has encouraged fundamental research in UAM to maintain the US global leadership in this field. The success of UAM will depend on the safe and robust multi-agent autonomy to scale up the operations to high-throughput urban air traffic. Learning-based techniques such as deep reinforcement learning and multi-agent reinforcement learning are developed to support planning and control for these eVTOL vehicles. However, there is a major challenge to provide theoretical safety and robustness guarantees for these learning-based neural network in-the-loop models in multi-agent autonomous UAM applications. In this project, the researchers will collaborate with committed government and industry partners on the use-case-inspired fundamental research, with a focus on promoting safety and reliability of AI, machine learning and autonomy in students with diverse backgrounds. The technical objectives of this project include (1) Safety and Robustness of Single-Agent Reinforcement Learning: in order to address the “safety critical” UAM challenge, the PIs plan the min-max optimization for single agent reinforcement learning to formally build sufficient safety margin, constrained reinforcement learning to formulate safety as physical constraints in state and action spaces, and the novel cautious reinforcement learning that uses variational policy gradient to plan the safest aircraft trajectory with minimum distributional risk; (2) Safety and Robustness of Multi-Agent Reinforcement Learning: in order to address the “heterogeneous agents and scalability” challenge, a novel federated reinforcement learning framework where a central agent coordinates with decentralized safe agents to improve traffic throughput while guaranteeing safety, and a scaling mechanism to accommodate a varying number of decentralized aircraft; (3) Safety and Robustness from Simulations to the Real World: in order to address the “high-dimensionality and environment uncertainty” challenge, the researchers will focus on the agents’ policy robustness under distribution shift and fast adaptation from simulation to the real world. Specifically, value-targeted model learning to incorporate domain knowledge such as the aircraft and environment physics, and a safe adaptation mechanism after the RL model is deployed online for flight testing or execution is planned.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该网络物理系统（CPS）项目旨在设计可扩展的多智能体规划和控制的理论和算法，以支持高吞吐量、不确定和动态环境中的安全关键型自主电动垂直起降飞机（UAM）是一种新兴的城市空中交通。电动垂直起降（eVTOL）飞机将在城市地区安全高效地运输乘客和货物的航空运输模式得到了白宫、美国国家工程院和美国国会的指导，鼓励了城市空中交通的基础研究。维持美国在这一领域的全球领先地位将取决于安全和强大的多智能体自主性，以将运营扩展到高吞吐量的城市空中交通，例如深度强化学习和多智能体强化。然而，在多智能体自主 UAM 应用中为这些基于学习的神经网络在环模型提供理论安全性和鲁棒性保证是一个重大挑战。项目中，研究人员将与有承诺的政府和该项目的技术目标包括（1）单智能体的安全性和鲁棒性。强化学习：为了解决“安全关键”的 UAM 挑战，PI 计划对单代理强化学习进行最小-最大优化，以正式建立足够的安全裕度，约束强化学习将安全性制定为状态和动作空间中的物理约束，和小说谨慎强化学习，使用变分策略梯度来规划具有最小分布风险的最安全的飞机轨迹；（2）多智能体强化学习的安全性和鲁棒性：为了解决“异构智能体和可扩展性”的挑战，一种新颖的联邦强化学习框架；其中中央代理与分散的安全代理协调以提高流量吞吐量，同时保证安全，以及适应不同数量的分散飞机的扩展机制（3）从模拟到安全性和鲁棒性；现实世界：为了应对“高维和环境不确定性”的挑战，研究人员将重点关注智能体在分布转移下的策略鲁棒性以及从模拟到现实世界的快速适应，具体来说，就是以价值为目标的模型学习。结合飞机和环境物理等领域知识，并计划在 RL 模型在线部署进行飞行测试或执行后建立安全适应机制。该奖项反映了 NSF 的法定使命，并通过使用基金会的知识评估进行评估，认为值得支持优点和更广泛的影响审查标准。