NSF-NSERC: SaTC: CORE: Small: Managing Risks of AI-generated Code in the Software Supply Chain

NSF-NSERC：SaTC：核心：小型：管理软件供应链中人工智能生成代码的风险

• 批准号: 2341206
• 负责人: Rachel Greenstadt
• 金额: $ 60万
• 依托单位: New York University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2024
• 资助国家: 美国
• 起止时间: 2024-06-01 至 2027-05-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2341206&HistoricalAwards=false
• 关键词: NSF; NSERC; SaTC; CORE; Small

Modern software is created by combining pre-existing software packages into a software product. This approach is enabled by the growing popularity of the Open-Source paradigm, where the source code of software packages is made available under licenses that allow reuse. This approach speeds up software development with significant economic benefits, but also creates the risk of inadvertently importing vulnerable code into critical software tools. The risk is further compounded by the increasing use of Artificial Intelligence (AI) tools for code generation in Open-Source development. These tools must be trained on enormous amounts of data, which is not always rigorously reviewed, and thus they may learn to generate vulnerable code. To make matters worse, malicious parties may actively inject malicious code in their training set. Unfortunately, all these issues are still poorly understood. This project aims at measuring and mitigating the risks emerging from AI-generated code in the software supply chain. It will investigate how prevalent the use of AI tools is, and characterize the security risks they entail. In doing so, it will address pressing economic and societal needs: AI promises to bring significant benefits to software development, but those can only be achieved if its risks are mitigated. The research outcomes will be disseminated through workshops and hackathons, and the results will become part of curriculum and courses. The work will benefit the open-source community by producing provenance tools to improve software supply chain security. The project is a collaboration with researchers from Canada with complementary expertise that provides additional resources to the project. Technically, the AI tools being investigated consist of various Large Language Models (LLM) for code generation. The threat model of interest is one where a developer inserts vulnerable LLM-generated code into a security-critical program, be it due to low-quality code generation or using a poisoned/backdoored LLM. This project consists of three thrusts, each addressing a research question relevant to the threat model: (i) how, and to what extent, LLM code can be distinguished from code written by humans; (ii) to what extent LLM code is already present in the supply chain, and what are its security implications; and (iii) to what extent poisoning attacks against LLM code generation can succeed in realistic conditions. In thrust (i), this project extends existing code stylometry techniques, until now used to distinguish human programmers, to the novel problem of distinguishing human- and LLM-generated code. In thrust (ii), the investigators conduct measurement studies of Open-Source software, generating empirical understanding of the presence and implications of LLM-generated code in the supply chain. Finally, thrust (iii) looks at the practical feasibility of code backdoors, and the effectiveness of automated reputation-based vetting as a defense.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

现代软件是通过将现有的软件包组合到软件产品中来创建的。这种方法是通过开源范例的日益普及而实现的，其中软件包的源代码在允许重用的许可证下提供。这种方法加快了软件开发速度，带来了显着的经济效益，但也带来了无意中将易受攻击的代码导入关键软件工具的风险。由于开源开发中越来越多地使用人工智能 (AI) 工具来生成代码，这一风险进一步加剧。这些工具必须接受大量数据的训练，而这些数据并不总是经过严格的审查，因此它们可能会学会生成易受攻击的代码。更糟糕的是，恶意方可能会主动在其训练集中注入恶意代码。不幸的是，所有这些问题仍然知之甚少。该项目旨在衡量和减轻软件供应链中人工智能生成的代码所带来的风险。它将调查人工智能工具的使用有多普遍，并描述它们所带来的安全风险。在此过程中，它将解决紧迫的经济和社会需求：人工智能有望为软件开发带来巨大的好处，但只有在降低风险的情况下才能实现这些好处。研究成果将通过研讨会和黑客马拉松传播，成果将成为课程和课程的一部分。这项工作将通过生产出处工具来提高软件供应链的安全性，从而使开源社区受益。 该项目是与加拿大研究人员合作的，他们具有互补的专业知识，为该项目提供了额外的资源。从技术上讲，正在研究的人工智能工具由用于代码生成的各种大型语言模型（LLM）组成。感兴趣的威胁模型是开发人员将易受攻击的 LLM 生成的代码插入到安全关键程序中的模型，无论是由于低质量的代码生成还是使用中毒/后门的 LLM。该项目由三个主旨组成，每个主旨都解决与威胁模型相关的研究问题：（i）如何以及在多大程度上将LLM代码与人类编写的代码区分开来； (ii) LLM 代码在多大程度上已存在于供应链中，以及其安全影响是什么； (iii) 针对 LLM 代码生成的中毒攻击在现实条件下能够成功到什么程度。在主旨 (i) 中，该项目将迄今为止用于区分人类程序员的现有代码风格测量技术扩展到区分人类和法学硕士生成的代码的新问题。在推力（ii）中，研究人员对开源软件进行了测量研究，对供应链中法学硕士生成的代码的存在和影响产生了实证理解。最后，推力 (iii) 着眼于代码后门的实际可行性，以及基于声誉的自动审查作为防御的有效性。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力优点和更广泛的评估进行评估，被认为值得支持。影响审查标准。