CAREER: Systematic Software Testing for Deep Learning Applications

职业：深度学习应用程序的系统软件测试

• 批准号: 1845893
• 负责人: Baishakhi Ray
• 金额: $ 53.01万
• 依托单位: Columbia University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-05-01 至 2025-04-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1845893&HistoricalAwards=false
• 关键词: CAREER; Systematic; Software; Testing; Deep

A paradigm shift is underway in software development, where decision making is increasingly shifting from hand-coded program logic to a reliance on Deep Learning (DL) --- popular applications of Speech Processing, Image Recognition, Robotics, etc. are using DL to implement their core components. Deep Neural Networks (DNNs), a widely used form of DL, is a key behind much of this progress. With such spectacular growth in traditional applications, DNNs and other DL technologies are also increasingly being used in safety-critical systems such as autonomous cars, medical diagnosis, malware detection, and aircraft collision avoidance systems. Such a wide adoption of DL techniques carries with it concerns about the reliability of these systems, as several high-profile instances of DL-based behavior have already been reported. Thus, it has become crucial to rigorously test these applications with realistic corner cases to ensure high reliability. However, due to the fundamental architectural differences between DL implementations such as DNNs and traditional software, existing software testing techniques do not apply to them in any obvious way. In fact, companies like Google, Tesla, etc. are increasingly confronting software testing challenges to ensure reliable and safe DL applications. Therefore, systematically testing DL-based software systems will be a significant step towards increasing safety and reliability of sensitive and safety-critical DL systems.This project will design, implement, and evaluate a novel software testing framework to assess the reliability of the Deep Learning applications and detect buggy behaviors during the application development and maintenance phase. In particular, the proposed framework will develop novel white-box testing strategies, realistic test-case generation techniques, and regression testing techniques to assess DL applications. A unique characteristic of the DL-based programming paradigm is that the end applications highly depend on the training data. Therefore, the research will build novel white-box testing strategies to evaluate both the model and the training data together as a whole system. In addition, this research will design and deploy techniques to generate new test cases that capture the real-world corner-case behavior where the DL applications may fail. The project will also investigate how any changes in data or model architecture can impact a pre-trained model in order to guide regression test case selection and prioritization process.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

软件开发领域正在发生范式转变，决策越来越多地从手工编码的程序逻辑转向依赖深度学习 (DL)——语音处理、图像识别、机器人等流行应用正在使用 DL 来实现他们的核心组件。深度神经网络 (DNN) 是一种广泛使用的深度学习形式，是这一进步的关键。随着传统应用的惊人增长，DNN 和其他深度学习技术也越来越多地应用于安全关键系统，例如自动驾驶汽车、医疗诊断、恶意软件检测和飞机防撞系统。深度学习技术的如此广泛采用也带来了对这些系统可靠性的担忧，因为已经报道了一些备受瞩目的基于深度学习的行为实例。因此，使用现实的极端情况严格测试这些应用程序以确保高可靠性变得至关重要。然而，由于深度神经网络（DNN）等深度学习实现与传统软件之间存在根本的架构差异，现有的软件测试技术并不能以任何明显的方式适用于它们。事实上，谷歌、特斯拉等公司越来越多地面临着软件测试挑战，以确保深度学习应用程序的可靠和安全。 因此，系统地测试基于深度学习的软件系统将是提高敏感和安全关键的深度学习系统的安全性和可靠性的重要一步。该项目将设计、实现和评估一种新颖的软件测试框架，以评估深度学习的可靠性应用程序并在应用程序开发和维护阶段检测错误行为。特别是，所提出的框架将开发新颖的白盒测试策略、现实的测试用例生成技术和回归测试技术来评估深度学习应用程序。基于深度学习的编程范式的一个独特特征是最终应用程序高度依赖于训练数据。因此，该研究将构建新颖的白盒测试策略，将模型和训练数据作为一个整体系统进行评估。此外，这项研究将设计和部署技术来生成新的测试用例，捕获深度学习应用程序可能失败的现实世界的极端情况行为。该项目还将调查数据或模型架构的任何变化如何影响预训练模型，以指导回归测试用例选择和优先级确定过程。该奖项反映了 NSF 的法定使命，并通过使用基金会的评估进行评估，被认为值得支持。智力价值和更广泛的影响审查标准。

项目成果

Learning Approximate Execution Semantics From Traces for Binary Function Similarity

• DOI: 10.1109/tse.2022.3231621
• 发表时间: 2023-04
• 期刊: IEEE Transactions on Software Engineering
• 影响因子: 7.4
• 作者: Kexin Pei;Zhou Xuan;Junfeng Yang;S. Jana;Baishakhi Ray

PMFuzz: test case generation for persistent memory programs

• DOI: 10.1145/3445814.3446691
• 发表时间: 2021-04
• 期刊: Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems
• 作者: Sihang Liu;Suyash Mahar;Baishakhi Ray;S. Khan

On Multi-Modal Learning of Editing Source Code

• DOI: 10.1109/ase51524.2021.9678559
• 发表时间: 2021-08
• 期刊: 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE)
• 作者: Saikat Chakraborty;Baishakhi Ray