CAREER: Securing Deep Reinforcement Learning

职业：保护深度强化学习

• 批准号: 2045948
• 负责人: Xinyu Xing
• 金额: $ 55.45万
• 依托单位: Pennsylvania State Univ University Park
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-10-01 至 2022-04-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2045948&HistoricalAwards=false
• 关键词: CAREER; Securing; Deep; Reinforcement; Learning

Like many other deep learning techniques, deep reinforcement learning is vulnerable to adversarial attacks. In reinforcement learning, an adversarial attack manipulates a reinforcement learning agent's sensory observation, flummoxing it. Recently, research has demonstrated that an adversarial attack could be even more practical. Instead of implicitly assuming an attacker has the full control to influence an agent's sensory system, the new type of attack presents an adversarial agent to manipulate the target agent's environment and thus trigger it to react in an undesired fashion. Compared with the kind of attack that alters the sensory observation, the new attack is more difficult to counteract. First, the methods (e.g., adversarial training) commonly used for robustifying other deep learning techniques are no longer suitable for deep reinforcement learning. Second, given a reinforcement learning agent, there are few technical approaches to scrutinizing the agent and unveiling its flaws. This project intends to address these two significant problems by integrating and expanding upon a series of technical approaches used in explainable AI, adversarial training, and formal verification in conjunction with program synthesis. The basic idea is first to learn an adversarial agent informed by explainable AI. Using this learned agent, we then unveil the weakness of target agents and adversarially train them accordingly. Through a robustness check, we evaluate the enhanced agents. If a strengthened agent fails the adversary-resistance check, we fall back on formal verification and program synthesis techniques. Using this unified solution, reinforcement learning model developers could identify the policy flaws of reinforcement learning agents and effectively remediate their weaknesses. This project will provide a stack of technical solutions to scrutinizing and robustifying deep reinforcement learning. If successful, the project will significantly advance the field of AI security (for adversarial training and adversarial policy learning) and contribute to the field of machine learning (for explainable AI and verified AI). Besides, this project has the potential to improve the security of reinforcement learning applications significantly.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

与许多其他深度学习技术一样，深度强化学习很容易受到对抗性攻击。在强化学习中，对抗性攻击会操纵强化学习代理的感官观察，使其混乱。最近，研究表明对抗性攻击可能更加实用。这种新型攻击不是隐含地假设攻击者拥有影响代理感知系统的完全控制权，而是提出了一个对抗性代理来操纵目标代理的环境，从而触发其以不期望的方式做出反应。与那种改变感官观察的攻击相比，新的攻击更难以抵抗。首先，通常用于强化其他深度学习技术的方法（例如对抗性训练）不再适合深度强化学习。其次，对于强化学习代理来说，几乎没有什么技术方法可以检查该代理并揭示其缺陷。 该项目旨在通过集成和扩展可解释人工智能、对抗性训练以及与程序合成相结合的形式验证中使用的一系列技术方法来解决这两个问题。基本思想是首先学习一个由可解释的人工智能告知的对抗代理。使用这个学习过的智能体，我们可以揭示目标智能体的弱点，并相应地进行对抗性训练。通过稳健性检查，我们评估增强的代理。如果强化的代理未能通过对手抵抗检查，我们就会依靠形式验证和程序综合技术。使用这个统一的解决方案，强化学习模型开发人员可以识别强化学习代理的策略缺陷并有效地纠正其弱点。该项目将提供一系列技术解决方案来审查和强化深度强化学习。如果成功，该项目将显着推进人工智能安全领域（针对对抗性训练和对抗性政策学习），并为机器学习领域（针对可解释人工智能和经过验证的人工智能）做出贡献。此外，该项目有可能显着提高强化学习应用的安全性。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。