CAREER: Systematic Software Testing for Deep Learning Applications

职业：深度学习应用程序的系统软件测试

• 批准号: 1845893
• 负责人: Baishakhi Ray
• 金额: $ 53.01万
• 依托单位: Columbia University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-05-01 至 2025-04-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1845893&HistoricalAwards=false
• 关键词: CAREER; Systematic; Software; Testing; Deep

A paradigm shift is underway in software development, where decision making is increasingly shifting from hand-coded program logic to a reliance on Deep Learning (DL) --- popular applications of Speech Processing, Image Recognition, Robotics, etc. are using DL to implement their core components. Deep Neural Networks (DNNs), a widely used form of DL, is a key behind much of this progress. With such spectacular growth in traditional applications, DNNs and other DL technologies are also increasingly being used in safety-critical systems such as autonomous cars, medical diagnosis, malware detection, and aircraft collision avoidance systems. Such a wide adoption of DL techniques carries with it concerns about the reliability of these systems, as several high-profile instances of DL-based behavior have already been reported. Thus, it has become crucial to rigorously test these applications with realistic corner cases to ensure high reliability. However, due to the fundamental architectural differences between DL implementations such as DNNs and traditional software, existing software testing techniques do not apply to them in any obvious way. In fact, companies like Google, Tesla, etc. are increasingly confronting software testing challenges to ensure reliable and safe DL applications. Therefore, systematically testing DL-based software systems will be a significant step towards increasing safety and reliability of sensitive and safety-critical DL systems.This project will design, implement, and evaluate a novel software testing framework to assess the reliability of the Deep Learning applications and detect buggy behaviors during the application development and maintenance phase. In particular, the proposed framework will develop novel white-box testing strategies, realistic test-case generation techniques, and regression testing techniques to assess DL applications. A unique characteristic of the DL-based programming paradigm is that the end applications highly depend on the training data. Therefore, the research will build novel white-box testing strategies to evaluate both the model and the training data together as a whole system. In addition, this research will design and deploy techniques to generate new test cases that capture the real-world corner-case behavior where the DL applications may fail. The project will also investigate how any changes in data or model architecture can impact a pre-trained model in order to guide regression test case selection and prioritization process.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

在软件开发中正在进行范式转变，在此过程中，决策越来越多地从手工编码的程序逻辑转移到对深度学习的依赖（DL）---流行语音处理，图像识别，机器人技术等的流行应用正在使用DL来实现其核心组件。深度神经网络（DNNS）是一种广泛使用的DL形式，是大部分进展的关键。随着传统应用中的如此壮观的增长，DNN和其他DL技术也越来越多地用于安全至关重要的系统，例如自动驾驶汽车，医疗诊断，恶意软件检测和飞机避免避免系统。 DL技术的这种广泛采用对这些系统的可靠性产生了关注，因为已经报道了几种基于DL的行为的备受瞩目的实例。因此，严格测试这些应用程序是至关重要的，以确保高可靠性。但是，由于DL实现（例如DNN和传统软件）之间的根本建筑差异，现有的软件测试技术并不以任何明显的方式适用于它们。实际上，诸如Google，Tesla等公司等公司越来越面临软件测试挑战，以确保可靠且安全的DL应用程序。 因此，系统测试基于DL的软件系统将是提高敏感和安全至关重要的DL系统的安全性和可靠性的重要一步。本项目将设计，实施和评估新颖的软件测试框架，以评估深度学习应用程序的可靠性并在应用程序开发阶段检测虫子行为的可靠性。特别是，所提出的框架将开发新颖的白盒测试策略，现实的测试案例生成技术和回归测试技术，以评估DL应用。基于DL的编程范式的独特特征是最终应用程序在很大程度上取决于培训数据。因此，该研究将建立新颖的白盒测试策略，以整个系统一起评估模型和培训数据。此外，这项研究将设计和部署技术，以生成新的测试用例，以捕获DL应用程序可能失败的现实世界角案例行为。该项目还将调查数据或模型体系结构的任何变化如何影响预先训练的模型，以指导回归测试案例选择和优先级别。该奖项反映了NSF的法定任务，并被认为是通过基金会的智力优点和更广泛影响的审查标准通过评估来通过评估来支持的。

项目成果

Learning Approximate Execution Semantics From Traces for Binary Function Similarity

• DOI: 10.1109/tse.2022.3231621
• 发表时间: 2023-04
• 期刊: IEEE Transactions on Software Engineering
• 影响因子: 7.4
• 作者: Kexin Pei;Zhou Xuan;Junfeng Yang;S. Jana;Baishakhi Ray

On Multi-Modal Learning of Editing Source Code

• DOI: 10.1109/ase51524.2021.9678559
• 发表时间: 2021-08
• 期刊: 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE)
• 作者: Saikat Chakraborty;Baishakhi Ray

PMFuzz: test case generation for persistent memory programs

• DOI: 10.1145/3445814.3446691
• 发表时间: 2021-04
• 期刊: Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems
• 作者: Sihang Liu;Suyash Mahar;Baishakhi Ray;S. Khan