SHF: Small: Automated Verification and Synthesis of Input Generators in Property-Based Testing Frameworks

SHF：小型：基于属性的测试框架中输入生成器的自动验证和合成

• 批准号: 2321680
• 负责人: Benjamin Delaware
• 金额: $ 59.78万
• 依托单位: Purdue University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-10-01 至 2026-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2321680&HistoricalAwards=false
• 关键词: SHF; Small; Automated; Verification; Synthesis

Testing is one of the most popular and effective ways to discover bugs in software, so that they can be fixed before a system is deployed. In recent years, automated testing has emerged as an important strategy for identifying software defects. Under this paradigm, developers specify the environment in which they expect their program to execute, and the behaviors it should exhibit in that environment. Given these constraints, automated testing frameworks attempt to systematically explore a program's behaviors by executing it in randomly generated environments consistent with the developer's characterization. Any unexpected behaviors are then reported back to the developer, so that they can diagnose and repair the underlying problem. Property-based testing is a popular automated testing approach that relies on handwritten programs, called generators, to construct the environment under which a target system is tested. Since they are also programs, generators may themselves have bugs which hamper the efficacy of automated testing. On the one hand, a generator may be unsound, constructing spurious environments that are inconsistent with the developer's requirements. An unsound generator results in a poor utilization of resources, as time is wasted looking for valid inputs. On the other hand, a generator may be incomplete, failing to produce valid environments. An incomplete generator lowers the level of assurance provided by the testing framework, as potentially faulty behaviors may be unexplored. Typically, developers rely on manual inspection and postmortem analysis of test runs to assess the soundness and completeness of a generator; not surprisingly, these approaches are error-prone and difficult to scale with generator complexity. The goal of this project is to develop new techniques that enable precise reasoning about the soundness and completeness of generators. The project's novelties are the development of new specification and reasoning frameworks, expressive type systems, and synthesis algorithms, specialized for the construction and validation of generators in property-based testing frameworks. Taken together, the project's impacts are a pathway to meaningfully strengthen the assurance provided by property based testing frameworks, resulting in an overall improvement in the quality of software validated using property-based testing.The project is comprised of three main thrusts. The first thrust considers specification frameworks and representations for characterizing the space of inputs produced by generators that are relevant to the systems under test. New specification formalisms capable of describing completeness properties, specifications that capture effectful properties, and quantitative specifications that describe the distributions and biases used to generate candidate inputs, will be developed in this thrust. The second thrust explores techniques for automatically verifying the correctness of user-defined generators. These approaches will focus on type-based verification techniques and will be influenced by the form and expressivity of the logical specifications developed in the first thrust. Finally, the third technical thrust investigates the complementary problem of directly synthesizing generators from the specifications developed in the first thrust, providing a correct-by-construction pathway for developers to automatically obtain high-quality generators.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

测试是发现软件中错误的最流行和有效方法之一，因此可以在部署系统之前将其修复。近年来，自动化测试已成为识别软件缺陷的重要策略。在此范式下，开发人员指定了他们期望其程序执行的环境以及在该环境中应表现出的行为。鉴于这些约束，自动测试框架试图通过在与开发人员的特征一致的随机生成环境中执行程序来系统地探索程序的行为。然后将任何意外行为报告给开发人员，以便他们可以诊断和修复潜在的问题。基于属性的测试是一种流行的自动测试方法，依赖于手写程序（称为发电机）来构建对目标系统进行测试的环境。由于它们也是程序，因此发电机本身可能会有障碍自动测试功效的错误。一方面，发电机可能是不健全的，构建了与开发人员要求不一致的虚假环境。不合能的发电机会导致资源的利用率不佳，因为时间浪费了寻找有效的输入。另一方面，发电机可能不完整，无法产生有效的环境。不完整的发电机降低了测试框架提供的保证水平，因为可能没有探索潜在的错误行为。通常，开发人员依靠手动检查和对测试运行后的验尸分析来评估发电机的健全性和完整性；毫不奇怪，这些方法容易出错，并且由于发电机的复杂性而难以扩展。该项目的目的是开发新技术，以实现有关发电机的健全性和完整性的精确推理。该项目的新颖性是开发新规范和推理框架，表达类型系统和综合算法，专门用于在基于物业的测试框架中构建和验证生成器。综上所述，该项目的影响是有意义地加强基于物业的测试框架提供的保证的途径，从而总体改善了使用基于物业的测试验证的软件质量。该项目由三个主要推力组成。 第一个推力考虑了指定框架和表示形式，以表征与正在测试的系统相关的发电机产生的输入空间。将在此推力中开发出能够描述完整性属性的新规范形式主义，能够描述完整性属性，捕获有效属性的规范以及描述用于生成候选输入的分布和偏见的定量规范。 第二个推力探索了自动验证用户定义发电机的正确性的技术。这些方法将集中在基于类型的验证技术上，并会受到第一个推力中开发的逻辑规范的形式和表现性的影响。最后，第三个技术推力研究了直接合成生成器与第一个推力中开发的规范合成的互补问题，为开发人员提供了正确的构建途径，以使开发人员自动获得高质量的发电机。该奖项反映了NSF的法规任务，并认为通过基金会的知识优点和广泛的crietia criter scritia criter criter criter criter criter criter criter criter criter criter criter criter critia criter criter criter criter critia critia criter critia critia criter criter criter crietia awne奖。