CAREER: Learning to Secure Cooperative Multi-Agent Learning Systems: Advanced Attacks and Robust Defenses

职业：学习保护协作多代理学习系统：高级攻击和强大的防御

• 批准号: 2146548
• 负责人: Zizhan Zheng
• 金额: $ 49.42万
• 依托单位: Tulane University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-06-01 至 2027-05-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2146548&HistoricalAwards=false
• 关键词: CAREER; Learning; Secure; Cooperative; Multi

Cooperative multi-agent learning (MAL), where multiple intelligent agents learn to coordinate with each other and with humans, is emerging as a promising paradigm for solving some of the most challenging problems in various security and safety-critical domains, including transportation, power systems, robotics, and healthcare. The decentralized nature of MAL systems and agents' exploration behavior, however, introduce new vulnerabilities unseen in standalone machine learning systems and traditional distributed systems. This project aims to develop a data-driven approach to MAL security that can provide an adequate level of protection even in the presence of persistent, coordinated, and stealthy malicious insiders or external adversaries. The main novelty of the project is to go beyond heuristics-based attack and defense schemes by incorporating opponent modeling and adaptation into security-related decision-making in a principled way. The project contributes to the emerging fields of science of security and trustworthy artificial intelligence via a cross-disciplinary approach that integrates cybersecurity, multi-agent systems, machine learning, and cognitive science. The interdisciplinary nature of this project also brings unique opportunities for both curriculum development and student training.Developing robust defenses for large-scale MAL systems faces fundamental challenges induced by the hidden behavioral patterns of malicious agents, the dynamics and uncertainty of the environment, and the necessity of protecting benign agents' local data in many privacy-sensitive settings. This project tackles the challenges by incrementally developing a (machine) theory of mind for adversarial decision-making in three research thrusts. The first thrust develops learning-based targeted and untargeted attacks against federated and decentralized machine learning systems. These attacks first infer a world model from publicly available data and then apply model-based reinforcement learning to identify an adaptive attack policy that can fully exploit the vulnerabilities of the systems. The second thrust investigates a proactive defense framework that combines adversarial training and local adaptation, utilizing the automated attack framework developed in the first thrust as a simulator of adversaries to obtain robust defenses. The third thrust studies security in cooperative multi-agent reinforcement learning systems by addressing a set of new challenges, including complicated interactions among agents, non-stationarity, and partial observability. The goal is to understand how malicious attacks and deceptions can prevent benign agents from reaching a socially preferred outcome and how accounting for a higher order of beliefs can help an agent (benign or malicious) in both fully cooperative and mixed-motive settings.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

合作的多机构学习（MAL），其中多个智能代理学会彼此和人类协调，它正成为解决各种安全和安全关键领域（包括运输，电力系统，机器人技术和医疗保健）中一些最具挑战性的问题的有希望的范式。然而，MAL系统和代理勘探行为的分散性质引入了独立的机器学习系统和传统分布式系统中看不见的新漏洞。该项目旨在开发一种以数据为基础的MAL安全方法，即使在存在持久，协调和隐秘的恶意内部人士或外部对手的情况下，也可以提供足够的保护水平。该项目的主要新颖性是通过以原则性的方式将对手建模和适应与安全相关的决策纳入基于启发式的攻击和防御方案。该项目通过跨学科的方法为新兴的安全和可信赖的人工智能科学领域做出了贡献，该方法可以整合网络安全，多机构系统，机器学习和认知科学。该项目的跨学科性质也为课程开发和学生培训带来了独特的机会。开发大型MAL系统的强大防御能力面临着由恶意代理的隐藏行为模式，环境的动态和不确定性的隐藏行为模式引起的基本挑战，以及在许多自私设置中保护良性良性代理人的本地数据的必要性。该项目通过在三个研究推力中逐步发展（机器）心理理论来应对挑战。第一个推力会开发基于学习的目标和非目标攻击，以针对联合和分散的机器学习系统。这些攻击首先从可公开的数据中推断出世界模型，然后应用基于模型的强化学习，以确定可以充分利用系统漏洞的自适应攻击策略。第二个推力研究了一个主动的防御框架，该框架结合了对抗性训练和当地适应，利用第一个推力在第一个推力中开发的自动攻击框架作为对手的模拟器，以获得强大的防御力。第三个推力通过应对一系列新的挑战，包括代理之间的复杂互动，非平稳性和部分可观察性来研究合作多代理增强学习系统中的安全性。目的是了解恶意的攻击和欺骗如何阻止良性代理人达到社会优先的结果，以及在完全合作和混合动力的环境中，更高的信念的考虑可以帮助代理人（良性或恶意）。该奖项反映了NSF的法定任务，并通过评估基金会的智力效果，并予以评估和广泛的范围。

项目成果

Robust Moving Target Defense Against Unknown Attacks: A Meta-reinforcement Learning Approach

• DOI: 10.1007/978-3-031-26369-9_6
• 发表时间: 2022
• 作者: Henger Li;Zizhan Zheng

Learning to Backdoor Federated Learning

• DOI: 10.48550/arxiv.2303.03320
• 发表时间: 2023-03
• 期刊: ArXiv
• 作者: Henger Li;Chen Wu;Senchun Zhu;Zizhan Zheng

Does Delegating Votes Protect Against Pandering Candidates? (Extended Abstract)

委托投票是否可以防止迎合候选人？

• 发表时间: 2023
• 期刊: International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS
• 作者: Sun, X.;Masur, J.;Abramowitz, B.;Mattei, N.;Zheng, Z.
• 通讯作者: Zheng, Z.

Learning to Attack Federated Learning: A Model-based Reinforcement Learning Attack Framework

• 发表时间: 2022
• 作者: Henger Li;Xiaolin Sun;Zizhan Zheng