Collaborative Research: SHF: Small: Feedback-Driven Mutation Testing for Any Language

合作研究：SHF：小型：任何语言的反馈驱动突变测试

• 批准号: 2129388
• 负责人: Claire Le Goues
• 金额: $ 25.55万
• 依托单位: Carnegie-Mellon University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-09-01 至 2024-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2129388&HistoricalAwards=false
• 关键词: Collaborative; Research; SHF; Small; Feedback; Collaborative; Research; SHF; Small; Feedback

Testing, validation, and verification are all central activities in programming and software engineering. Unfortunately, existing techniques for testing remain inadequate for finding and eliminating key vulnerabilities before software deployment -- even the most critical modern software is rife with security vulnerabilities and defects that ultimately cost the economy billions of dollars annually in lost productivity and compromised data. A technique known as "mutation testing" has been researched since the 1970s; it aims to help software engineers improve their tests and their software at the same time, by automatically adding bugs to a program and checking whether the test suite can detect them. Although in theory this technique is extremely effective for improving software quality, there are several fundamental factors that prevent it from being widely used in practice: it is difficult and time-consuming to use, and the tools that exist for it cannot all handle the diversity of program languages that are deployed in modern software systems. This project will tackle these challenges and allow this important technique to be used to improve quality of real-world software by developing efficient tools that can apply mutation testing to programs written in any language; prioritize the output of the tools to reduce the amount of time and effort needed to make maximal use of them; and incorporate user feedback into the technique to maximize testing efficiency. The project will be evaluated on real-world open source software like the Linux kernel, and build on the researchers' previous collaborations to substantially improve program and test effort quality on critical real-world software.The core problem this project aims to address is making program mutants practical in nonresearch settings, in a way that meets the needs of developers and test engineers, by making it possible for someone creating or enhancing a test suite, or developing code and test suite in tandem, to (1) use "just enough" mutation testing for their needs, maximizing benefit gained in exchange for work performed, and (2) to work in any programming language without worrying about the quality of tool support provided for mutation testing, and without sacrificing the ease of understanding of source-based mutants, while easily adding custom mutation operators that target their specific software development task. This project aims to adapt the Furthest-Point-First metric previously used in fuzzer bug triaging to the problem of maximizing the novelty of mutants examined by a user, in order to make it possible to quickly discover unkilled mutants that expose serious defects in a testing or verification effort. However, novelty alone is not sufficient: feedback-driven mutation testing must also help users avoid inconsequential, equivalent mutants, kill mutants high in the dominance hierarchy, and (most importantly) incorporate user feedback. If a user marks a mutant as inconsequential, or equivalent, or (especially) high impact, then that information must be used to inform the ranking of future mutants as well. In order to make such an approach maximally valuable, this project also proposes to improve the state-of-the-art in source-level multilingual mutant generation, allowing users to easily generate mutants for new programming languages, or even for custom DSLs that are part of a specific project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

测试，验证和验证都是编程和软件工程中的中心活动。不幸的是，现有的测试技术在软件部署前发现和消除关键漏洞仍然不足 - 即使是最关键的现代软件也是有安全漏洞和缺陷，这些漏洞和缺陷最终使经济损失了数十亿美元的生产率和损害数据。自1970年代以来，已经研究了一种称为“突变测试”的技术。它旨在通过自动将错误添加到程序中并检查测试套件是否可以检测到它们，以帮助软件工程师同时改善其测试和软件。尽管从理论上讲，该技术对于提高软件质量非常有效，但有几个基本因素可以阻止其在实践中被广泛使用：它很困难且耗时，而存在的工具不能全部处理在现代软件系统中部署的程序语言的多样性。该项目将解决这些挑战，并允许该重要的技术通过开发可以将突变测试应用于以任何语言编写的程序应用的有效工具来改善现实世界软件的质量；优先考虑工具的输出，以减少最大程度地使用它们所需的时间和精力；并将用户反馈纳入该技术，以最大化测试效率。该项目将在现实世界中的开源软件（例如Linux内核）上进行评估，并基于研究人员以前的合作，以实质上改善计划和测试关键现实世界软件上的质量。该项目旨在解决该项目的核心问题是使程序突变体在非研究中实用，从而使某人在不得不满足某人的需求或增强某人的需求中，以使其能够实现或增强某人的需求，以使某人在某人的需求中进行测试，或者在某人的需求中进行测试或增强范围，或者在某人的范围内进行效果，或者在某人的范围内进行效果，并且可以使某人努力且能够努力，并且可以在某人的需求中进行效率或增强范围。 （1）使用“足够的”突变测试来满足其需求，最大限度地提高福利以换取所执行的工作，而不必担心为突变测试提供的工具支持质量而工作，而不必牺牲对基于源的突变体的易于理解，同时又轻松地添加了针对其特定软件开发任务的定制突变操作员。该项目旨在调整以前用于模糊错误的最较高的量子标准，以最大程度地提高用户检​​查突变体的新颖性的问题，以便能够快速发现在测试或验证工作中暴露出严重缺陷的无渗透突变体。但是，仅新颖性就不够：反馈驱动的突变测试还必须帮助用户避免无关紧要的，等效的突变体，在优势层次结构中杀死高较高的突变体，并且（最重要的是）包含用户反馈。 如果用户将突变体标记为无关紧要的或同等的，或者（尤其是）高影响力，则必须使用该信息来告知未来突变体的排名。为了使这种方法具有最大价值，该项目还建议改善来源级别的多语言突变体生成的最先进，使用户可以轻松地为新的编程语言生成突变体，甚至是针对特定项目的自定义DSL的突变体。该奖项是NSF的法定任务，并反映了通过评估的依据，该奖项反映了众所周知的依据，并已被评估范围众所周知。