SHF: Small: AI Model Debugging by Analyzing Model Internals with Python Program Analysis

SHF：小：通过 Python 程序分析分析模型内部结构进行 AI 模型调试

• 批准号: 1910300
• 负责人: Xiangyu Zhang
• 金额: $ 50万
• 依托单位: Purdue University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1910300&HistoricalAwards=false
• 关键词: SHF; Small; AI; Model; Debugging

Artificial Intelligence (AI) in general, and especially Machine Learning (ML), is playing an increasingly important role in many aspects of our life. AI/ML model engineering (e.g., model training, tuning, and maintenance) is hence becoming an essential part of modern software-systems engineering. Just like software inevitably contains bugs and software debugging is a key step in software development, AI/ML models may have buggy behaviors, and thus model debugging will be a critical step in software engineering. AI model bugs may lead to undesirable consequences such as low model accuracy and vulnerabilities to security attacks, which substantially hinder the application of AI models, especially in safety-critical areas. The current practice of AI model debugging mainly focuses on tuning parameters and providing additional training data. However, it does not try to diagnose the root cause from the observable symptoms and then repair accordingly. There are no general debugging tools that work for a large set of models. The current state of practice is to perform tedious and redundant tests on individual implementations and even at a per-training-session level. AI models, especially neural network models, are essentially programs (e.g., in Python) that compute state-variable values, called neuron activations, through multiple program phases (called layers). The values of neurons in a layer are computed from those of the previous layer through matrix multiplication and application of an activation function, which is basically a threshholding function to determine if values will be used in the computation of the next layer. The project will develop techniques that consider AI models as programs with specific semantics such that AI model debugging can substantially benefit from analyzing these programs and their execution states. As such, the substantial experience of software debugging that is built up by the software-engineering and program-analysis community over decades of intensive R&D can be leveraged to build general and novel AI model debuggers. Just like software-debugging tools help developers to inspect program states and identify root causes, these AI model debuggers will allow data engineers to compare internal neuron-activation values of correctly classified and mis-classified cases to identify root cause features that lead to misclassification, by providing guidance in selecting more inputs to improve model behavior related to these features.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

人工智能 (AI)，尤其是机器学习 (ML)，在我们生活的许多方面发挥着越来越重要的作用。 因此，AI/ML 模型工程（例如模型训练、调整和维护）正在成为现代软件系统工程的重要组成部分。就像软件不可避免地包含错误并且软件调试是软件开发的关键步骤一样，AI/ML 模型也可能存在错误行为，因此模型调试将成为软件工程的关键步骤。人工智能模型的错误可能会导致模型精度低、易受安全攻击等不良后果，从而严重阻碍人工智能模型的应用，特别是在安全关键领域。目前AI模型调试的实践主要集中在调整参数和提供额外的训练数据上。但是，它不会尝试从可观察到的症状诊断根本原因，然后进行相应的修复。没有适用于大量模型的通用调试工具。当前的实践状态是对单个实现甚至在每个训练会话级别执行繁琐且冗余的测试。 AI 模型，尤其是神经网络模型，本质上是通过多个程序阶段（称为层）计算状态变量值（称为神经元激活）的程序（例如，Python）。一层中神经元的值是通过矩阵乘法和应用激活函数从前一层的神经元值计算出来的，激活函数基本上是一个阈值函数，用于确定值是否将用于下一层的计算。该项目将开发将人工智能模型视为具有特定语义的程序的技术，以便人工智能模型调试可以从分析这些程序及其执行状态中获益匪浅。因此，软件工程和程序分析社区在数十年的密集研发中积累的丰富的软件调试经验可以用来构建通用和新颖的人工智能模型调试器。就像软件调试工具帮助开发人员检查程序状态并识别根本原因一样，这些人工智能模型调试器将允许数据工程师比较正确分类和错误分类案例的内部神经元激活值，以识别导致错误分类的根本原因特征，通过提供选择更多输入的指导来改进与这些功能相关的模型行为。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Deep Feature Space Trojan Attack of Neural Networks by Controlled Detoxification

通过受控解毒对神经网络进行深度特征空间木马攻击

• DOI: 10.1609/aaai.v35i2.16201
• 发表时间: 2020-12-21
• 作者: Siyuan Cheng;Yingqi Liu;Shiqing Ma;X. Zhang
• 通讯作者: X. Zhang

ABS: Scanning Neural Networks for Back-doors by Artificial Brain Stimulation

ABS：通过人工大脑刺激扫描神经网络是否存在后门

• 发表时间: 2019-11
• 期刊: Proceedings of the 26th ACM Conference on Computer and Communications Security (CCS 2019
• 作者: Yingqi Liu; Wen
• 通讯作者: Wen

On-the-fly Improving Performance of Deep Code Models via Input Denoising

通过输入去噪即时提高深度代码模型的性能

• DOI: 10.1109/ase56229.2023.00166
• 发表时间: 2023-08-19
• 期刊: 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)
• 作者: Zhao Tian;Junjie Chen;Xiangyu Zhang
• 通讯作者: Xiangyu Zhang

RULER: Discriminative and Iterative Adversarial Training for Deep Neural Network Fairness

RULER：深度神经网络公平性的判别性和迭代对抗性训练

• 发表时间: 2022-01
• 期刊: Proceedings of the 2022 ACM SIGSOFT International Symposium on the Foundations of Software Engineering
• 作者: Tao, Guanhong;Sun, Weisong;Han, Tingxu;Fang, Chunrong;Zhang, Xiangyu
• 通讯作者: Zhang, Xiangyu

PELICAN: Exploiting Backdoors of Naturally Trained Deep Learning Models In Binary Code Analysis

PELICAN：在二进制代码分析中利用自然训练的深度学习模型的后门

• 发表时间: 2024-09-13
• 作者: Zhuo Zhang;Guanhong Tao;Guangyu Shen;Shengwei An;Qiuling Xu;Yingqi Liu;Yapeng Ye;Yaoxuan Wu
• 通讯作者: Yaoxuan Wu