SHF: Small: Automated Unit Test Generation using Large Language Models

SHF：小型：使用大型语言模型自动生成单元测试

基本信息

批准号：
2307742
负责人：
Frank Tip
金额：
$ 60万
依托单位：
Northeastern University
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2023
资助国家：
美国
起止时间：
2023-07-01 至 2026-06-30
项目状态：
未结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=2307742&HistoricalAwards=false
关键词：
SHF Small Automated Unit Test

项目摘要

Large Language Models (LLMs) are systems that rely on machine learning techniques to solve queries that are stated using natural language. To interact with an LLM, a user provides a textual description, commonly referred to as a "prompt", of the problem to be solved. In response to receiving a prompt, an LLM generates one or more textual results that represent solutions to the problem. LLMs have recently become very popular for applications such as translation and chatbots. Furthermore, LLMs are starting to be adopted for applications in software development such as automatic code completion in Integrated Development Environments used by programmers. This project explores the use of LLMs for automatically testing software. To this end, tests are generated by providing LLMs with prompts containing code fragments of the software under test and artifacts associated with the software such as documentation and usage examples. The project will result in improved testing of software, thereby improving its quality. Moreover, the project will unburden developers from the chore of writing tests manually, thereby increasing their productivity. The developed automatic test generation techniques based on Large Language Models (LLMs) will feature a feedback-directed, iterative approach. In each iteration, an LLM is given a prompt containing the signature of a function under test, along with usage examples, test framework code, and results obtained from previously generated tests. The resulting completions consist of candidate tests, which are checked for syntactic validity and executed to check if they pass or fail. Analysis of the execution behavior of failing tests will direct the construction of refined prompts. The project will target both dynamically typed and statically typed programming languages. The work will be evaluated by comparing the generated test suites against those produced by state-of-the-art test generation tools. All developed test generation tools will be made available as open-source software for others to use and build on, and results will be disseminated via publications. Societal benefits will follow from improvements in software quality enabled by the adoption of the developed techniques.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

大型语言模型 (LLM) 是依靠机器学习技术来解决使用自然语言表述的查询的系统。为了与法学硕士进行交互，用户需要提供要解决的问题的文本描述，通常称为“提示”。为了响应收到的提示，法学硕士会生成一个或多个代表问题解决方案的文本结果。法学硕士最近在翻译和聊天机器人等应用领域变得非常流行。此外，法学硕士开始应用于软件开发中的应用，例如程序员使用的集成开发环境中的自动代码完成。该项目探讨了如何使用法学硕士来自动测试软件。为此，通过向法学硕士提供包含被测软件的代码片段以及与软件相关的工件（例如文档和使用示例）的提示来生成测试。该项目将改进软件测试，从而提高其质量。此外，该项目将使开发人员摆脱手动编写测试的苦差事，从而提高他们的生产力。基于大型语言模型（LLM）开发的自动测试生成技术将采用反馈导向的迭代方法。在每次迭代中，LLM 都会收到一个提示，其中包含被测函数的签名、使用示例、测试框架代码以及从先前生成的测试中获得的结果。最终的完成结果包括候选测试，这些测试将检查语法有效性并执行以检查它们是否通过。对失败测试的执行行为的分析将指导细化提示的构建。该项目将针对动态类型和静态类型编程语言。将通过将生成的测试套件与最先进的测试生成工具生成的测试套件进行比较来评估工作。所有开发的测试生成工具都将作为开源软件提供给其他人使用和构建，结果将通过出版物传播。通过采用所开发的技术而实现的软件质量的提高将带来社会效益。该奖项反映了 NSF 的法定使命，并且通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。