Collaborative Research: SLES: Verifying and Enforcing Safety Constraints in AI-based Sequential Generation

合作研究：SLES：验证和执行基于人工智能的顺序生成中的安全约束

• 批准号: 2331967
• 负责人: Huan Zhang
• 金额: $ 26万
• 依托单位: University of Illinois at Urbana-Champaign
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-10-01 至 2026-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2331967&HistoricalAwards=false
• 关键词: Collaborative; Research; SLES; Verifying; Enforcing

Artificial intelligence (AI) has achieved transformative impacts on various complex real-world challenges. Among its applications, sequential data are prevalent in many critical usages of AI when it directly engages with its users. Self-driving cars rely on AI to process sequences of sensor data from cameras and radars, and make a sequence of real-time decisions to ensure safe driving. Healthcare monitoring systems use AI to analyze sequences of patient health data, such as blood pressure, heart rate, and others, to detect anomalies and predict potential health issues. Chatbots utilize AI to understand natural language and generate safe, fair, and appropriate text responses as sequences of words and sentences. The sequential data produced by AI make its behavior hard to characterize because of the complex dependencies within the sequence, and a careless application of AI in these scenarios may lead to harmful consequences, such as a collision of an autonomous vehicle or the generation of biased or toxic texts. This project aims to study the safety of AI under scenarios with sequential data, provide assurance for its behavior in mission-critical environments, and ensure AI-based sequential generation can adhere to safety constraints and social norms. Ultimately, this research will help with reducing unexpected AI failures, preventing bias and discrimination in AI technologies, aligning AI systems with human values and societal norms, and building up public trust for AI-enabled applications.The technical contributions of this project consist of three thrusts. The first thrust develops a formal verification framework for assuring the safety of AI models for sequential generation tasks with rigorous mathematical guarantees. It includes a series of innovative verification algorithms for bound propagation and branch-and-bound for general non-linear functions involved in sequential generation models. These new verification methods will be integrated into the alpha-beta-CROWN neural network verifier, a well-known open-source toolbox developed by investigators. The second thrust involves training and inference algorithms that ensure sequential generation models comply with specified safety constraints, with a unique probabilistic framework that decomposes a safety constraint into action-level components and enforces them at each generation step. This approach can be integrated with model training to improve the safety performance of sequential generation models using posterior regularization techniques. Lastly, the third thrust aims to integrate the formal verification and constrained generation components above and apply them to three important real-world applications: safety of text generation, safety and stability of controlled systems, and robust AI-generated text detectors. This project will also result in tools to the broader AI community, including the alpha-beta-CROWN neural network verifier, and the shared data and benchmarks developed to evaluate the safety of sequential generation models.This project is supported by a partnership with the NSF and Open Philanthropy.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

人工智能（AI）已经对各种复杂的现实世界挑战产生了变革性的影响。在其应用程序中，当它直接与用户互动时，顺序数据在AI的许多关键用法中都普遍存在。自动驾驶汽车依靠AI来处理来自相机和雷达的传感器数据序列，并做出一系列实时决策以确保安全驾驶。医疗保健监测系统使用AI来分析患者健康数据的序列，例如血压，心率等，以检测异常并预测潜在的健康问题。聊天机器人利用AI来理解自然语言，并将安全，公平和适当的文本响应作为单词和句子序列。 AI产生的顺序数据使其行为很难表征，因为序列内的复杂依赖性以及在这些情况下AI的粗心应用可能会导致有害后果，例如自动驾驶汽车的碰撞或产生有偏见的或有毒的文本。该项目旨在通过顺序数据研究AI的安全性，为其在任务至关重要的环境中的行为提供保证，并确保基于AI的顺序生成可以遵守安全限制和社会规范。最终，这项研究将有助于减少意外的AI失败，防止AI技术中的偏见和歧视，使AI系统具有人为价值和社会规范的AI系统，并建立对AI-Spable应用程序的公共信任。该项目的技术贡献由三个推力组成。第一个推力开发了一个正式的验证框架，以确保AI模型的安全性，以使用严格的数学保证来进行顺序生成任务。它包括一系列用于结合传播的创新验证算法，以及与顺序生成模型有关的一般非线性功能的分支和分支。这些新的验证方法将集成到Alpha-Beta-crown神经网络验证器中，这是研究人员开发的众所周知的开源工具箱。第二个推力涉及训练和推理算法，以确保顺序生成模型符合指定的安全限制，并具有独特的概率框架，可将安全性约束分解为动作级别的组件并在每个一代步骤中执行它们。可以将这种方法与模型训练集成，以改善使用后正规化技术的顺序生成模型的安全性能。最后，第三个推力旨在整合上面的正式验证和约束的生成组件，并将其应用于三个重要的现实世界应用：文本生成的安全性，受控系统的安全性和稳定性以及强大的AI生成的文本检测器。该项目还将为更广泛的AI社区提供工具，包括Alpha-Beta-crown神经网络验证器，并开发了共享的数据和基准，以评估顺序生成模型的安全性。该项目得到了与NSF的合作关系和开放的慈善事业的伙伴关系。这些奖项反映了NSF的范围和众所周知的Intortial Merit，该奖项通过智力的构建范围来进行了良好的影响，该奖项是由良好的构建构建的依据。 标准。