Collaborative Research: SHF: Medium: Causal Performance Debugging for Highly-Configurable Systems

合作研究：SHF：中：高度可配置系统的因果性能调试

• 批准号: 2107405
• 负责人: Baishakhi Ray
• 金额: $ 37.3万
• 依托单位: Columbia University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2107405&HistoricalAwards=false
• 关键词: Collaborative; Research; SHF; Medium; Causal; Collaborative; Research; SHF; Medium; Causal

Software performance is critical for most software systems to achieve scale and limit operating costs and energy consumption. As modern software systems, such as big data and machine-learning systems, are increasingly built by composing many reusable infrastructure components and deployed on distributed and heterogeneous hardware, developers have powerful tools and abstractions at their fingertips, and as a result face immense configuration complexity. Software and hardware need to be selected and configured carefully to achieve high performance for a given system and task. Unfortunately, in practice, performance faults and misconfigurations are common, where a system performs much worse than expected, not achieving its mission or simply wasting cost and energy. In large configuration spaces, end-users and developers face severe challenges in understanding and fixing performance faults by changing software configuration, changing hardware deployment, or modifying the software's code itself. Current approaches that model system performance by analyzing correlations among performance measurements and options are slow and may produce misleading results, obfuscating the actual causes of performance faults. Even if they can fix the problem, most of them cannot explain why (1) the obtained configurations are the real cause of the problem, and (2) a user/developer should consider the proposed recommendations. In both cases, the lack of explainability is a big issue. The project is intended to initiate a paradigm shift in today's testing and debugging methodology for complex, highly configurable systems, thereby positively impacting a broad range of industrial sectors relying on complex, highly configurable systems. Specifically, the project contributes to substantial energy savings and reduced carbon emissions, especially for the many big-data and machine-learning systems that operate at a massive scale. Finally, the research is providing valuable training for involved students from diverse backgrounds in research and generating high-quality researchers and practitioners for society. This project develops and evaluates foundations and tools for a causal approach to performance modeling and performance debugging. This project introduces the new concept of causal performance models that are learned using causal structure learning by intervening over configuration options and observing system performance regarding (multiple) performance objectives, rather than just analyzing correlations. Causal models enable causal inference and counterfactual reasoning for numerous tasks, including debugging performance faults and misconfigurations. Based on a solid technical foundation of causal modeling and extensive experience with performance modeling for configurable systems, this project develops innovations in three thrusts: (1) It designs and refines a causal modeling approach for software performance of systems composed of multiple configurable components, using innovations in sampling strategies, code analysis, compositional reasoning, and transfer learning to build accurate causal models efficiently. (2) It develops and evaluates user-facing tool support, based on causal models, to help users select well-performing configurations for their specific tasks and hardware and resolve misconfiguration faults with configuration changes, highlighting the (causal) performance impact of configuration decisions and providing a Pareto analysis of involved tradeoffs. (3) It develops and evaluates developer-facing tool support to foster code-level debugging and documentation. Finally, all contributions are being evaluated end-to-end with developers on real performance faults, showing how both users and developers benefit from causal models and related tools.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

软件性能对于大多数软件系统至关重要，以实现规模并限制运营成本和能源消耗。由于现代软件系统（例如大数据和机器学习系统）越来越多地通过组成许多可重复使用的基础架构组件并部署在分布式和异质硬件上，因此开发人员的触手可及的工具和抽象具有强大的工具和抽象。需要仔细选择和配置软件和硬件，以实现给定系统和任务的高性能。不幸的是，在实践中，性能错误和配置错误很常见，因为系统的性能比预期的要差得多，而不是实现其任务或仅仅浪费成本和精力。在大型配置空间中，最终用户和开发人员通过更改软件配置，更改硬件部署或修改软件的代码本身来理解和修复性能错误时面临严重的挑战。当前的方法通过分析性能测量和选项之间的相关性来对系统性能进行建模，这可能会产生误导性结果，从而掩盖了性能故障的实际原因。即使他们可以解决问题，大多数人也无法解释为什么（1）获得的配置是问题的真正原因，并且（2）用户/开发人员应考虑提出的建议。在这两种情况下，缺乏解释性都是一个大问题。该项目旨在为复杂的，高度可配置的系统的当今测试和调试方法启动范式转移，从而积极影响依靠复杂，高度可配置的系统的广泛工业领域。具体而言，该项目有助于节省大量能源并减少碳排放，尤其是对于许多大规模运行的大数据和机器学习系统。最后，这项研究为来自研究的不同背景的学生提供了宝贵的培训，并为社会创造了高质量的研究人员和从业人员。该项目开发并评估了基础和工具，以实现因果方法进行绩效建模和绩效调试的方法。该项目介绍了因果绩效模型的新概念，这些概念是通过介入配置选项并观察有关（多个）性能目标的系统性能，而不仅仅是分析相关性来学习的。因果模型为许多任务提供了因果推理和反事实推理，包括调试绩效错误和配置错误。 Based on a solid technical foundation of causal modeling and extensive experience with performance modeling for configurable systems, this project develops innovations in three thrusts: (1) It designs and refines a causal modeling approach for software performance of systems composed of multiple configurable components, using innovations in sampling strategies, code analysis, compositional reasoning, and transfer learning to build accurate causal models efficiently. （2）它基于因果模型来开发和评估面向用户的工具支持，以帮助用户选择具有特定任务和硬件的表现良好的配置，并通过配置变化解决错误的配置故障，突出显示（因果关系）的绩效影响（因果关系）对配置决策的影响并提供了涉及交易的帕累托分析。 （3）它开发和评估面向开发人员的工具支持，以促进代码级调试和文档。最后，所有贡献都在与开发人员的实际绩效错误有关的端到端评估，展示了用户和开发人员如何从因果模型和相关工具中受益。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的智力优点评估来支持的，并具有更广泛的影响标准。