CAREER: Enhanced Reliability and Efficiency of Software Regression Testing in the Presence of Flaky Tests

职业：在存在不稳定测试的情况下增强软件回归测试的可靠性和效率

• 批准号: 2338287
• 负责人: Wing Lam
• 金额: $ 61.85万
• 依托单位: George Mason University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2024
• 资助国家: 美国
• 起止时间: 2024-10-01 至 2029-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2338287&HistoricalAwards=false
• 关键词: CAREER; Enhanced; Reliability; Efficiency; Software

Software is usually developed in a continuous development and integration process that incorporates incremental changes leading to successive releases of the software, where each release undergoes rigorous software testing to check whether recent code changes had broken existing functionalities. This process, known as regression testing, is widely used in software development practice. A major problem in the generation of test cases is the presence of flaky tests: tests that non-deterministically pass or fail on the same version of the code. Failures from flaky tests can mislead developers about their recent changes, waste developers’ time, and reduce developers’ trust in software testing. Many software development organizations have reported that flaky tests are one of their biggest problems, because they confound assurance goals. This project aims to improve the reliability and efficiency of regression testing in the presence of flaky tests. It will produce tools that aim to be efficient and effective at resolving the inherent nondeterminism. The work focuses on (1) reducing the cost of flaky-test detection and debugging techniques by predicting important test properties, (2) developing new techniques to predict flakiness-related properties, (3) speeding up and reducing the resources needed by regression testing, (4) developing new techniques to systematically detect flaky tests, and (5) reducing the flakiness in Android user interface testing. The project will also produce curriculum for education and training on the topic of programming in the face of nondeterminism, and will work with industry to transfer technology. The research on flakiness will move from the typical, black-box approaches to a new level for detecting, debugging, and fixing through novel, white-box and learning-based approaches. The approach will use static and dynamic analyses to compute state pollution, which may affect test flakiness based on the test execution order. The work involves combinatorial design theory to improve the efficiency of order-dependent test detection. Special attention will be paid to flaky tests in graphical user interfaces using record-and-replay and test input generation. Test coverage computations, which can be used to predict whether a code change will affect the test's output, will use a machine learning approach. The work will result in tool implementations and large-scale evaluations in open source and proprietary environments.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

软件通常是在连续开发和集成过程中开发的，该过程结合了增量更改，导致软件的成功发行，其中每个版本都经过严格的软件测试，以检查最近的代码更改是否破坏了现有功能。这个被称为回归测试的过程被广泛用于软件开发实践中。生成测试用例的一个主要问题是存在片状测试：在同一版本的代码上，非确定性通过或失败的测试。 Flaky测试的故障可能会使开发人员误以为他们最近的变化，浪费开发人员的时间以及减少开发人员对软件测试的信任。许多软件开发组织报告说，片状测试是他们最大的问题之一，因为它们会混淆保证目标。该项目旨在在存在片状测试的情况下提高回归测试的可靠性和效率。它将产生旨在提高固有的非确定性的工具。 The work focuses on (1) reducing the cost of flaky-test detection and debugging techniques by predicting important test properties, (2) developing new techniques to predict flakiness-related properties, (3) speeding up and reducing the resources needed by regression testing, (4) developing new technologies to systematically detect flaky tests, and (5) reducing the flakiness in Android user interface testing.该项目还将在面对不确定性的情况下为编程主题提供教育和培训课程，并将与行业合作转移技术。关于片状的研究将从典型的黑盒方法转变为通过新颖的，白盒和基于学习的方法来检测，调试和修复的新级别。该方法将使用静态和动态分析来计算状态污染，这可能会根据测试执行顺序影响测试片状。这项工作涉及组合设计理论，以提高依赖订单依赖性测试检测的效率。使用记录和重新播放和测试输入生成，将特别注意在图形用户界面中的片状测试。可以使用机器学习方法来预测代码更改是否会影响测试的输出的测试覆盖计算。这项工作将导致工具实施和开源环境中的大规模评估。该奖项反映了NSF的法定任务，并使用基金会的知识分子优点和更广泛的影响审查标准来诚实地认为通过评估诚实地获得了支持。