SHF: Small: Distribution-aware Testing for Neural Networks

SHF：小型：神经网络的分布感知测试

• 批准号: 2129824
• 负责人: Matthew Dwyer
• 金额: $ 49.85万
• 依托单位: University of Virginia Main Campus
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2129824&HistoricalAwards=false
• 关键词: SHF; Small; Distribution; aware; Testing

Machine learning aims to learn the definition of a software implementation from a set of examples. For instance, from a large set of images from the forward facing camera in an automobile such approaches can learn to detect when a pedestrian is near the road. This method is attractive because traditional software development approaches have a hard time with this kind of problem specification -- consider the challenge of precisely describing what a pedestrian looks like given all the variations in their clothing, what they are carrying, their location and pose, whether they are partially obscured, partially lit, etc. Using machine learning, new capabilities can thus be used to deploy potentially better systems. The challenge is how to determine that these new capabilities work as expected. For traditional software, decades of research has yielded sophisticated frameworks for testing software. This project will adapt the strategies from that body of research to test machine learning models. In this way the project aims to improve the safety and quality of advanced machine-learning-based software systems, which will have positive impacts on people and organizations who depend on those systems.The project will exploit learned models of a target data distribution to drive testing of neural networks that are trained on samples from that distribution. The methods that are developed will exploit generative models that exhibit high-precision and recall of the target distribution and that have a well-defined mathematical structure to their latent space. By targeting this latent space, distribution-aware neural-network testing techniques can leverage its reduced dimensionality, relative to the neural network's input domain, to assess the adequacy of a given test suite and to generate new and valuable tests. In addition, the project will develop a rigorous mutation-based framework for evaluating the fault-detection effectiveness of neural network testing methods and to employ that framework in evaluating the benefits of distribution-aware testing methods.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

机器学习旨在从一组示例中学习软件实现的定义。 例如，从汽车中的一大批图像中，这种方法可以学会检测行人何时在道路附近。 这种方法很有吸引力，因为传统的软件开发方法在这种问题规范上很困难 - 考虑到鉴于他们的服装，携带的东西，其位置和姿势的所有变化，确切描述行人的外观的挑战，无论是部分被掩盖，部分点燃的，是否部分掩盖了等等。使用机器学习，因此可以使用机器学习，因此可以使用新的功能来部署潜在的潜在的更好的系统。 挑战是如何确定这些新功能按预期工作。 对于传统软件，数十年的研究产生了用于测试软件的复杂框架。 该项目将调整研究机构的策略来测试机器学习模型。 通过这种方式，该项目旨在提高基于机器的软件系统的安全性和质量，这将对依赖这些系统的人员和组织产生积极的影响。该项目将利用目标数据分布的学习模型来推动对来自该分布的样本进行培训的神经网络测试。 开发的方法将利用对目标分布的高精度和回忆的生成模型，并且对其潜在空间具有明确的数学结构。 通过针对这一潜在空间，相对于神经网络的输入域，分布感知的神经网络测试技术可以利用其降低的维度，以评估给定测试套件的适当性并生成新的和有价值的测试。 此外，该项目将开发一个基于严格的突变框架，用于评估神经网络测试方法的断层检测有效性，并采用该框架来评估分布意见测试方法的好处。该奖项反映了NSF的法定任务，并通过评估该基金会的知识绩效和广泛的影响来评估NSF的法定任务，并被视为值得进行的支持。

项目成果

Input Distribution Coverage: Measuring Feature Interaction Adequacy in Neural Network Testing

• DOI: 10.1145/3576040
• 发表时间: 2022-12
• 期刊: ACM Transactions on Software Engineering and Methodology
• 影响因子: 4.4
• 作者: Swaroopa Dola;Matthew B. Dwyer;M. Soffa

Sibyl: Improving Software Engineering Tools with SMT Selection

• DOI: 10.1109/icse48619.2023.00184
• 发表时间: 2023-01-01
• 期刊: 2023 IEEE/ACM 45TH INTERNATIONAL CONFERENCE ON SOFTWARE ENGINEERING, ICSE
• 作者: Leeson，Will;Dwyer，Matthew B.;Filieri，Antonio
• 通讯作者: Filieri，Antonio