CAREER: Towards Gray-Fault Tolerant Cloud through Harnessing and Enhancing System Observability

职业：通过利用和增强系统可观测性迈向灰色容错云

• 批准号: 1942794
• 负责人: Peng Huang
• 金额: $ 60.95万
• 依托单位: Johns Hopkins University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-07-01 至 2023-04-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1942794&HistoricalAwards=false
• 关键词: CAREER; Towards; Gray; Fault; Tolerant

Cloud systems are the crucial infrastructure to many services existing today. Ensuring cloud software runs continuously without disruptions is both vital and challenging. Decades of research have developed mature techniques to detect and mask faults in distributed systems. But these techniques often use a simple model that assumes a system component either works or completely stops. Numerous real-world cloud incidents, however, suggest that production cloud systems frequently experience gray failures---a degraded operational mode in which a system component appears to be working but is in fact severely impaired. Gray failures cannot be effectively dealt with by current solutions. The overall objective of this proposal is to develop a holistic approach to detect, pinpoint and diagnose gray failures in production cloud systems. To realize the objective, four synergistic research activities are proposed. Specifically, the project conducts a study on real-world gray failure cases in popular distributed systems, measure and characterize the observability of existing systems. The project then designs a novel hybrid analysis that automatically inserts report-generation hooks across the whole systems stack to harness observability for detecting gray failures. To pinpoint the culprit component, this project further proposes algorithms to infer causality from the collected observations. Lastly, this project designs a runtime checking framework for increasing observability and online diagnosis of gray failures. Gray failures are a common cause of cloud service outages, resulting in significant financial loss. This project can effectively improve our understandings of gray failures and help detect and debug gray failures to reduce their impact on the ubiquitous cloud infrastructures. Software is moving to be more distributed with increasing subtle failure modes. Observability, fault detection, and localization are critical skills for this paradigm shift but are rarely covered in the existing curriculum. This project addresses this educational gap through curriculum development and student training. This project also promotes Computer Science education to underrepresented Baltimore high school students by organizing workshops in partnership with a non-profit organization, Code in the Schools, for local high school students to showcase cloud and system failure concepts.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

云系统是当今许多现有服务的关键基础设施。确保云软件持续运行而不中断既重要又具有挑战性。数十年的研究已经开发出成熟的技术来检测和屏蔽分布式系统中的故障。但这些技术通常使用一个简单的模型，假设系统组件要么工作要么完全停止。然而，大量现实世界的云事件表明，生产云系统经常遇到灰色故障——一种降级的操作模式，其中系统组件看似正常工作，但实际上受到严重损害。目前的解决方案无法有效处理灰色故障。该提案的总体目标是开发一种整体方法来检测、查明和诊断生产云系统中的灰色故障。为了实现这一目标，提出了四项协同研究活动。具体来说，该项目对流行分布式系统中的现实灰色故障案例进行研究，测量和表征现有系统的可观察性。然后，该项目设计了一种新颖的混合分析，可以自动在整个系统堆栈中插入报告生成挂钩，以利用可观察性来检测灰色故障。为了查明罪魁祸首，该项目进一步提出了从收集的观察结果中推断因果关系的算法。最后，该项目设计了一个运行时检查框架，以提高灰色故障的可观测性和在线诊断能力。灰色故障是云服务中断的常见原因，会导致重大的财务损失。该项目可以有效提高我们对灰色故障的认识，帮助检测和调试灰色故障，减少其对无处不在的云基础设施的影响。随着微妙故障模式的增加，软件正变得更加分布式。可观察性、故障检测和定位是这种范式转变的关键技能，但现有课程很少涵盖。该项目通过课程开发和学生培训来解决这一教育差距。该项目还与非营利组织 Code in the Schools 合作组织研讨会，向当地高中生展示云和系统故障概念，从而促进巴尔的摩高中学生的计算机科学教育。该奖项反映了 NSF 的法定使命和通过使用基金会的智力价值和更广泛的影响审查标准进行评估，该项目被认为值得支持。

项目成果

Demystifying and Checking Silent Semantic Violations in Large Distributed Systems

揭秘并检查大型分布式系统中的无声语义违规

• 发表时间: 2022-07
• 期刊: 16th USENIX Symposium on Operating Systems Design and Implementation
• 作者: Lou, Chang;Jing, Yuzhuo;Huang, Peng
• 通讯作者: Huang, Peng

Predictive and Adaptive Failure Mitigation to Avert Production Cloud VM Interruptions

预测性和适应性故障缓解以避免生产云虚拟机中断

• 发表时间: 2020-12-31
• 作者: Sebastien Levy;R;olph Yao;olph;Youjiang Wu;Yingnong Dang;Peng Huang;Zheng Mu;Pu Zhao;Tarun Ramani;N. Govindaraju;Xukun Li;Qingwei Lin;Gil Lapid Shafriri;Murali Chintalapati
• 通讯作者: Murali Chintalapati

Understanding and dealing with hard faults in persistent memory systems

理解和处理持久内存系统中的硬故障

• DOI: 10.1145/3447786.3456252
• 发表时间: 2021-04-21
• 期刊: Proceedings of the Sixteenth European Conference on Computer Systems
• 作者: Brian Choi;R. Burns;Peng Huang
• 通讯作者: Peng Huang

Understanding, Detecting and Localizing Partial Failures in Large System Software

理解、检测和定位大型系统软件中的部分故障

• 发表时间: 2020-02
• 期刊: Proceedings of the 17th USENIX Symposium on Networked Systems Design
• 作者: Lou, Chang;Huang, Peng;Smith, Scott
• 通讯作者: Smith, Scott

RESIN: A Holistic Service for Dealing with Memory Leaks in Production Cloud Infrastructure

RESIN：用于处理生产云基础设施中内存泄漏的整体服务

• DOI: 10.1016/j.jelechem.2020.114884
• 发表时间: 2024-09-13
• 期刊: Journal of Electroanalytical Chemistry
• 影响因子: 4.5
• 作者: Chang Lou;Congwei Chen;Peng Huang;Yingnong Dang;Si Qin;Xinsheng Yang;Xukun Li;Qingwei Lin;Murali Chintalapati;Microsoft Azure
• 通讯作者: Microsoft Azure