CNS Core: Small: Testing and detecting software upgrade failures in data-intensive distributed systems

CNS 核心：小型：测试和检测数据密集型分布式系统中的软件升级故障

• 批准号: 2300562
• 负责人: Yongle Zhang
• 金额: $ 60万
• 依托单位: Purdue University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-10-01 至 2026-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2300562&HistoricalAwards=false
• 关键词: CNS; Core; Small; Testing; detecting

In the current big data era, Internet services are often built on top of data-intensive distributed systems. Such distributed systems have to go through frequent software upgrade as vendors need to add new features, improve performance, and deploy patches. With the rise of continuous deployment in the industry, the frequency of distributed system software upgrade could reach thousands of deployments in a single day in a major Internet company. Unfortunately, distributed systems could experience upgrade failures – failures happen during software upgrade. These failures often have large-scale impact as upgrade is performed on the entire system. They are typically mitigated in the production environment with canary deployment, which slowly rollout updates from a small scale to the entire cluster and downgrade if a failure is encountered. However, canary deployment easily takes hours and creates a dilemma between safe and fast upgrade. In addition, many upgrade failures have persistent impact and cannot be easily resolved by downgrading. Despite the severe consequence of upgrade failures and challenges faced by production mitigation techniques, there are no existing testing and program analysis techniques that focus on testing and analyzing the distributed system upgrade procedure systematically. This work proposes to develop such techniques optimized to detect upgrade failures in early stages through exploring the effectiveness of unique properties of the distributed system software upgrade procedure. Data-intensive distributed systems deployed in public or private clouds are nowadays a cornerstone of many critical computing systems. The proposed techniques should dramatically improve the reliability of data-intensive distributed systems during upgrade and, consequently, reduce service disruptions and improve the availability of cloud systems. In addition, improved reliability of the upgrade procedure will lead to more timely feedbacks about new features in production, which is critical for developers’ productivity and the quality of the resulting software.In this project we plan to (1) implement differential testing between two standard distributed system upgrade procedures – full-stop upgrade and rolling upgrade, (2) explore utilizing source code difference between versions to design differential test oracles, feedback metrics, and input mutation strategies, that are specially tuned to trigger and detect upgrade failures, (3) design static program analysis guided by source code difference to detect data format incompatibilities between versions, and (4) validate the testing and detection techniques proposed through direct experimentation on real-world data-intensive distributed systems. The proposed fault localization and static analysis techniques will reduce the valuable time and effort that developers spend on root cause diagnosis, which is extremely challenging for bugs in distributed systems. All products of the project will be open sourced to ensure a widespread impact.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

在当前的大数据时代，互联网服务通常建立在数据密集型分布式系统之上。这样的分布式系统必须经常进行软件升级，因为供应商需要添加新功能，提高性能和部署补丁。随着行业中连续部署的不断增加，分布式系统软件升级的频率可能会在一家主要的互联网公司中一天内到达数千个部署。不幸的是，分布式系统可能会体验到升级失败 - 在软件升级期间发生故障。由于对整个系统进行升级，这些故障通常会产生大规模影响。它们通常会在生产环境中通过金丝雀部署来缓解它们，这些部署会缓慢地推出更新到整个集群，并在遇到故障时降级。但是，金丝雀部署很容易花费数小时，并在安全和快速升级之间造成了困境。此外，许多升级失败都具有持久的影响，并且不能通过降级来轻松解决。尽管升级失败和生产缓解技术面临的挑战的严重后果，但尚无现有的测试和程序分析技术，这些技术专注于系统地测试和分析分布式系统升级程序。这项工作建议通过探索分布式系统软件升级过程的独特属性的有效性来开发优化的技术，以在早期阶段检测升级失败。如今，部署在公共或私有云中的数据密集型分布式系统是许多关键计算系统的基石。提出的技术应大大提高升级过程中数据密集型分布式系统的可靠性，从而减少服务中断并改善云系统的可用性。 In addition, improved reliability of the upgrade procedure will lead to more timely feedbacks About new features in production, which is critical for developers’ productivity and the quality of the resulting software.In this project we plan to (1) implement differential testing between two standard distributed system upgrade procedures – full-stop upgrade and rolling upgrade, (2) explore utilizing source code difference between versions to design differential test oracles, feedback metrics, and input突变策略是专门调整以触发和检测升级失败的，（3）设计静态程序分析以源代码差异为指导，以检测版本之间的数据格式不兼容，并且（（4）通过直接实验对现实世界中数据密集的分布式系统进行直接实验，验证了测试和检测技术。拟议的故障定位和静态分析技术将减少开发人员在根本原因诊断上花费的宝贵时间和精力，这对于分布式系统中的错误极为挑战。该项目的所有产品将被开放为确保宽度影响。该奖项反映了NSF的法定任务，并使用基金会的知识分子优点和更广泛的影响评估标准，认为通过评估被认为是珍贵的支持。