CPS: Medium: Collaborative Research: Provably Safe and Robust Multi-Agent Reinforcement Learning with Applications in Urban Air Mobility

CPS：中：协作研究：可证明安全且鲁棒的多智能体强化学习及其在城市空中交通中的应用

• 批准号: 2312092
• 负责人: Peng Wei
• 金额: $ 40万
• 依托单位: George Washington University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-06-01 至 2026-05-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2312092&HistoricalAwards=false
• 关键词: CPS; Medium; Collaborative; Research; Provably

This Cyber-Physical Systems (CPS) project aims at designing theories and algorithms for scalable multi-agent planning and control to support safety-critical autonomous eVTOL aircraft in high-throughput, uncertain and dynamic environments. Urban Air Mobility (UAM) is an emerging air transportation mode in which electrical vertical take-off and landing (eVTOL) aircraft will safely and efficiently transport passengers and cargo within urban areas. Guidance from the White House, the National Academy of Engineering, and the US Congress has encouraged fundamental research in UAM to maintain the US global leadership in this field. The success of UAM will depend on the safe and robust multi-agent autonomy to scale up the operations to high-throughput urban air traffic. Learning-based techniques such as deep reinforcement learning and multi-agent reinforcement learning are developed to support planning and control for these eVTOL vehicles. However, there is a major challenge to provide theoretical safety and robustness guarantees for these learning-based neural network in-the-loop models in multi-agent autonomous UAM applications. In this project, the researchers will collaborate with committed government and industry partners on the use-case-inspired fundamental research, with a focus on promoting safety and reliability of AI, machine learning and autonomy in students with diverse backgrounds. The technical objectives of this project include (1) Safety and Robustness of Single-Agent Reinforcement Learning: in order to address the “safety critical” UAM challenge, the PIs plan the min-max optimization for single agent reinforcement learning to formally build sufficient safety margin, constrained reinforcement learning to formulate safety as physical constraints in state and action spaces, and the novel cautious reinforcement learning that uses variational policy gradient to plan the safest aircraft trajectory with minimum distributional risk; (2) Safety and Robustness of Multi-Agent Reinforcement Learning: in order to address the “heterogeneous agents and scalability” challenge, a novel federated reinforcement learning framework where a central agent coordinates with decentralized safe agents to improve traffic throughput while guaranteeing safety, and a scaling mechanism to accommodate a varying number of decentralized aircraft; (3) Safety and Robustness from Simulations to the Real World: in order to address the “high-dimensionality and environment uncertainty” challenge, the researchers will focus on the agents’ policy robustness under distribution shift and fast adaptation from simulation to the real world. Specifically, value-targeted model learning to incorporate domain knowledge such as the aircraft and environment physics, and a safe adaptation mechanism after the RL model is deployed online for flight testing or execution is planned.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该网络物理系统（CPS）项目旨在设计用于可扩展的多代理计划和控制的理论和算法，以在高通量，不确定和动态的环境中支持安全至关重要的自主EVTOL飞机。城市空气流动性（UAM）是一种新兴的航空运输模式，其中电垂直起飞和降落（EVTOL）飞机将安全有效地在城市地区运输密码和货物。白宫，美国国家工程学院和美国国会的指导鼓励UAM的基础研究，以维持美国在该领域的全球领导。 UAM的成功将取决于安全，强大的多代理自治权，以扩大运营，以扩大到高通量城市空中交通。开发了基于学习的技术，例如深厚的增强学习和多代理增强学习，以支持这些EVTOL车辆的计划和控制。但是，在多代理自动企业UAM应用程序中的环路模型中，为这些基于学习的中性网络提供理论安全性和鲁棒性保证存在一个重大挑战。在该项目中，研究人员将与有忠诚的政府和行业合作伙伴合作就用例启发的基础研究进行合作，重点是促进具有潜水员背景学生的AI，机器学习和自治的安全性和可靠性。该项目的技术目标包括（1）单位加强学习的安全性和鲁棒性：为了解决“安全关键” UAM挑战，PIS计划了对单一剂量强化学习的最小值优化，从分配风险； （2）多代理强化学习的安全性和鲁棒性：为了解决“异质的代理和可伸缩性”挑战，这是一个新型联合加固学习框架，中央代理与分散的安全剂坐在交通范围的情况下，以改善交通吞吐量，并确保安全的安全机制，以适应一个vary量的飞机数量； （3）从模拟到现实世界的安全性和鲁棒性：为了应对“高维和环境不确定性”的挑战，研究人员将重点关注代理商的稳健性，并从分配转移和从模拟到现实世界的快速适应。具体来说，有价值的模型学习以合并诸如飞机和环境物理的领域知识，并计划在网上部署RL模型以进行飞行测试或执行之后的安全适应机制。该奖项反映了NSF的法定任务，并通过评估该基金会的知识绩效和广泛的影响来评估NSF的法定任务。