CNS Core: Small: Intelligent Fault Injection to Expose and Reproduce Production-Grade Bugs in Cloud Systems

CNS 核心：小型：智能故障注入以暴露和重现云系统中的生产级错误

• 批准号: 2317698
• 负责人: Peng Huang
• 金额: $ 50万
• 依托单位: Regents of the University of Michigan - Ann Arbor
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-01-01 至 2024-11-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2317698&HistoricalAwards=false
• 关键词: CNS; Core; Small; Intelligent; Fault

Failures of production distributed systems are costly. Despite extensive efforts on testing distributed systems, many bugs remain difficult to find in testing even when a system is tested with appropriate input. This is because these bugs are triggered by the unique faulty events in the production environment. Fault injection has been proposed to simulate faults during testing with the goal of catching such bugs. However, existing solutions treat the target systems as a black box and only inject simple faults using random choices. Production failures are often caused by bugs that require complex, system-specific faults at careful timing to trigger.This project takes a holistic approach to address the fundamental limitations in current fault injection testing. The project develops special compiler support to enable the injection of system-specific faults at a fine granularity with precise control. To efficiently explore the large fault injection space and expose bugs, this project designs new fault injection decision algorithms and machine learning methods. A new adaptive method further analyzes production execution traces to quickly reproduce fault-induced failures in offline environment. Bugs in production distributed systems have resulted in substantial financial losses to society. The new fault injection techniques developed in this project will help effectively catch a wide range of production-grade bugs in large distributed systems and improve the availability of cloud services. This project will closely engage with developers in the open-source community to improve the distributed systems code quality and testing practice.The software artifact this project develops will be open sourced and available at https://github.com/OrderLab. The project results, including paper publications, technical reports, and presentations will be made available for free download and be maintained for at least five years beyond the completion of the project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

生产分布式系统的故障代价高昂。尽管在测试分布式系统方面付出了巨大的努力，但即使使用适当的输入来测试系统，在测试中仍然很难发现许多错误。这是因为这些错误是由生产环境中特有的故障事件触发的。人们提出了故障注入来模拟测试过程中的故障，目的是捕获此类错误。 然而，现有的解决方案将目标系统视为黑匣子，并且仅使用随机选择注入简单的故障。 生产故障通常是由错误引起的，这些错误需要在仔细的时间触发复杂的、系统特定的故障。该项目采用整体方法来解决当前故障注入测试中的基本限制。 该项目开发了特殊的编译器支持，以便能够通过精确控制以细粒度注入系统特定的故障。为了有效地探索庞大的故障注入空间并暴露错误，该项目设计了新的故障注入决策算法和机器学习方法。一种新的自适应方法进一步分析生产执行轨迹，以在离线环境中快速重现故障引起的故障。生产分布式系统中的错误给社会造成了巨大的经济损失。该项目开发的新故障注入技术将有助于有效捕获大型分布式系统中的各种生产级错误，并提高云服务的可用性。该项目将与开源社区的开发人员密切合作，以提高分布式系统代码质量和测试实践。该项目开发的软件工件将开源并可在 https://github.com/OrderLab 上获取。项目成果，包括纸质出版物、技术报告和演示文稿，将可供免费下载，并在项目完成后至少保留五年。该奖项反映了 NSF 的法定使命，经评估认为值得支持利用基金会的智力优势和更广泛的影响审查标准。