CIF: Small: Adversarially Robust Reinforcement Learning: Attack, Defense, and Analysis

CIF：小型：对抗性鲁棒强化学习：攻击、防御和分析

• 批准号: 2232907
• 负责人: Lifeng Lai
• 金额: $ 50万
• 依托单位: University of California-Davis
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-07-01 至 2026-06-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2232907&HistoricalAwards=false
• 关键词: CIF; Small; Adversarially; Robust; Reinforcement

In order to develop trustworthy machine-learning systems, it is essential to understand the potential vulnerabilities of existing learning algorithms and then develop corresponding mitigation strategies. Reinforcement learning (RL), a framework for control-theoretic problems that makes decisions over time within uncertain environments, has many applications in a variety of scenarios, such as recommendation systems, autonomous driving, and finance and business management, to name a few. In modern industry-scale applications of RL models, action decisions, reward- and state-signal collection, and policy iterations are normally implemented in distributed networks. When data packets containing reward signals and action decisions are transmitted through the network, an attacker can intercept and modify these packets to implement adversarial attacks. As RL models are being increasingly deployed in safety-critical and security-related applications, there is a pressing need to understand the effects of potential adversarial attacks on these applications.In this project, the investigator aims to address the following questions: 1) Should decisions made by RL agents be trusted?; 2) Can an adversary mislead RL agents?; and 3) How to design RL algorithms that are robust to adversarial attacks? While many existing works address adversarial attacks on supervised learning models, the understandings of vulnerabilities of RL models and their corresponding mitigation strategies are less complete, partially due to the significant differences between online RL and supervised learning. In particular, compared with the supervised-learning setting, the design and analysis of attack/defense mechanisms for RL models have to handle challenges such as long-term rewards, no access to future data, and unknown dynamics. The goal of this project is to overcome these challenges and make initial attempts to answer the questions raised above. In particular, this project aims to: 1) systematically investigate potential vulnerabilities of RL models and algorithms, 2) develop robust RL algorithms that can mitigate the impacts of adversarial attacks, and 3) analyze the benefit/cost of these mitigation strategies.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

为了开发值得信赖的机器学习系统，有必要了解现有学习算法的潜在漏洞，然后制定相应的缓解策略。强化学习 (RL) 是一种控制理论问题的框架，可在不确定的环境中随着时间的推移做出决策，在各种场景中都有许多应用，例如推荐系统、自动驾驶、财务和商业管理等。在现代工业规模的强化学习模型应用中，行动决策、奖励和状态信号收集以及策略迭代通常在分布式网络中实现。当包含奖励信号和行动决策的数据包通过网络传输时，攻击者可以拦截并修改这些数据包以实施对抗性攻击。随着强化学习模型越来越多地部署在安全关键和安全相关的应用程序中，迫切需要了解潜在的对抗性攻击对这些应用程序的影响。在这个项目中，研究人员旨在解决以下问题：1）应该RL 代理做出的决策值得信任吗？ 2) 对手能否误导 RL 智能体？； 3）如何设计对对抗性攻击具有鲁棒性的强化学习算法？虽然许多现有的工作解决了监督学习模型的对抗性攻击，但对强化学习模型的漏洞及其相应的缓解策略的理解还不够完整，部分原因是在线强化学习和监督学习之间存在显着差异。特别是，与监督学习环境相比，强化学习模型的攻击/防御机制的设计和分析必须应对诸如长期奖励、无法访问未来数据和未知动态等挑战。该项目的目标是克服这些挑战并初步尝试回答上述问题。具体而言，该项目旨在：1）系统地研究 RL 模型和算法的潜在漏洞，2）开发可以减轻对抗性攻击影响的强大 RL 算法，3）分析这些缓解策略的收益/成本。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。