SHF: Medium: Collaborative Research: ANACIN-X: Analysis and modeling of Nondeterminism and Associated Costs in eXtreme scale applications

SHF：中：协作研究：ANACIN-X：极端规模应用中的非确定性和相关成本的分析和建模

• 批准号: 1900888
• 负责人: Michela Taufer
• 金额: $ 91.57万
• 依托单位: University of Tennessee Knoxville
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-08-01 至 2025-07-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1900888&HistoricalAwards=false
• 关键词: SHF; Medium; Collaborative; Research; ANACIN

Nondeterminism (i.e., the properties of a scientific application to exhibit different behaviors in numerical results and execution patterns during multiple executions) is an increasingly entrenched property of high performance computing (HPC) applications as the scientific community is moving their simulations on larger and highly heterogeneous computing systems. Nondeterminism can drastically increase the cost of scientific reproducibility in terms of developer time and computational resources, debugging applications when moving from a smaller to a larger scale or from one platform to another, and ensuring fault-tolerance when executions may need to recover from a system fault. These three challenges can ultimately compromise the amount and quality of scientific discovery through computer simulations. Tools for addressing aspects of the nondeterministic problem have emerged, including Record-and-replay (R&R) techniques that monitor and record changes in program states over one execution (i.e., the recorded execution) of an application; and reproduce those changes, and thus, the behavior of the application during a subsequent execution (i.e., the replayed execution). However, these tools impose overheads on the underlying application and thus present HPC users with the problem of balancing tool utility against tool overhead. HPC users may opt to not use the tool at all rather than deal with unpredictable overheads. This project supports HPC users by modeling the relationship between application nondeterminism and variability in tool overhead, and uses this knowledge to identify hot spots in terms of tool cost as well as regions in executions that trigger nondeterministic behaviors in the applications. The aim of the project is to model nondeterministic executions by determining points (motif) of nondeterminism in executions of HPC applications and to apply the motif modeling with R&R techniques, to study the cost on R&R techniques of certain motifs. The outcome of this project impacts four communities of application developers with the identification of sources of unintended nondeterminism and their management; the HPC research community working on fault-tolerance, resilience, and reproducibility at exascale; data center administrators who use evaluation tools for and with application developers; and educators and trainers in resource constrained environments to promote HPC without the need of accessing high-end, expensive computers.This project advances the study of nondeterministic HPC applications by studying the recording costs of Record-and-replay (R&R) tools and by defining strategy so that these tools can scale to the exascale domain. In addition to the more commonly studied factors of time and memory overhead, the project integrates power usage in the modeling. The project relies on graph theory to develop expressive and scalable graph-based representations of the dependencies between events in a program, and develops algorithms to identify motifs in the graph that indicate points of nondeterminism. These motifs are applied to quantify the associated costs of nondeterminism, including developing metrics to measure dissimilarities between different executions, modeling the costs of recording executions and assessing the overhead of recordings. Based on these motifs, work on this project generates ?fingerprints? (i.e., a holistic characterization of how and where nondeterminism manifests during the application executions) of real world HPC applications including N-Body problems (e.g., simulating particle, atomic, and planetary interactions); (2) Graph analytics (e.g., Graph500 benchmark); (3) Bioinformatics (e.g., mpiBLAST); and (4) Task-based data analysis application (e.g., WordCount, Join, Octree Clustering on top of MapReduce Over MPI frameworks). The fingerprints illuminate previously-overlooked similarities between the nondeterminism that manifests across multiple classes of applications and allow users to probe the relationship between process communication patterns, the motifs of the actual resulting executions, and the regions of those executions in which tool overhead accumulates for nondeterministic HPC applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

非确定性（即，在多次执行过程中在数值结果和执行模式中表现出不同行为的科学应用的特性）是高性能计算应用程序（HPC）应用程序越来越根深蒂固的属性，因为科学界正在将其模拟在较大且高度异构的计算系统上移动。非确定性可以大大提高开发人员时间和计算资源的科学可重复性成本，从较小规模或从一个平台转移到另一个平台时进行调试应用程序，并在执行可能需要从系统故障中恢复时，确保耐受性容忍。这三个挑战最终可以通过计算机模拟损害科学发现的数量和质量。 已经出现了用于解决非确定问题方面的工具，包括记录和复制（R＆R）技术，这些技术在一个应用程序的一个执行（即记录的执行）中监视和记录程序状态的变化；并重现这些更改，因此，在随后的执行过程中应用程序的行为（即重播执行）。但是，这些工具在基础应用程序上强加了开销，因此向HPC用户呈现了平衡工具实用程序与工具开销的问题。 HPC用户可能会选择根本不使用该工具，而不是处理不可预测的开销。该项目通过建模应用程序非确定性和工具间接费用之间的关系来支持HPC用户，并使用此知识来识别刀具成本以及执行区域的热点，这些区域在应用程序中触发非确定行为的执行区域。该项目的目的是通过确定HPC应用程序执行中的非确定性点（主题）来对非确定性执行进行建模，并使用R＆R技术应用主题建模，以研究某些主题的R＆R技术成本。该项目的结果影响了四个应用程序开发人员社区，并确定了意外的非确定性及其管理的来源； HPC研究社区在Exascale致力于耐断层，弹性和可重复性。为应用程序开发人员使用评估工具的数据中心管理员；以及资源约束环境中的教育工作者和培训师，以促进HPC而无需访问高端，昂贵的计算机。该项目通过研究记录和复兴（R＆R）工具的录制成本（R＆R）工具的录制成本并定义策略，以使这些工具可以扩展到Exascale域。除了更常见的时间和内存开销因素外，该项目还将功率使用集成在建模中。该项目依靠图理论来开发程序中事件之间依赖关系的表达性和可扩展图的表示，并开发算法以识别图表中指示非确定点的图案。这些图案用于量化非确定性的相关成本，包括开发指标以衡量不同执行之间的差异，对记录执行的成本进行建模并评估录音的开销。基于这些图案，该项目的工作会生成“指纹”吗？ （即，在应用程序执行过程中非确定性表现出的方​​式和何处的整体表征）现实世界中的HPC应用程序，包括n体问题（例如，模拟粒子，原子和行星相互作用）； （2）Graph Analytics（例如Graph500基准）; （3）生物信息学（例如Mpiblast）; （4）基于任务的数据分析应用程序（例如WordCount，JOIN，OCTREE在MAPREDUCE之上与MPI框架之上的聚类）。指纹阐明了以前被忽视的非确定性之间的相似性，在多个类别的应用程序之间表现出来，并允许用户探测过程沟通模式之间的关系，实际结果执行的实际执行主题，实际执行的区域以及这些执行的区域以及工具架设的工具额外累积的HPC应用程序的累积。基金会的智力优点和更广泛的影响评论标准。

项目成果

A Research-Based Course Module to Study Non-determinism in High Performance Applications

用于研究高性能应用中的非确定性的研究型课程模块

• DOI: 10.1109/ipdpsw55747.2022.00067
• 发表时间: 2022
• 期刊: 2022 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW
• 作者: Bell, Patrick;Suarez, Kae;Fossum, Barbara;Chapp, Dylan;Bhowmick, Sanjukta;Taufer, Michela
• 通讯作者: Taufer, Michela

ANACIN-X: A software framework for studying non-determinism in MPI applications

ANACIN-X：用于研究 MPI 应用中的非确定性的软件框架

• DOI: 10.1016/j.simpa.2021.100151
• 发表时间: 2021
• 期刊: Software Impacts
• 作者: Bell, Patrick;Suarez, Kae;Chapp, Dylan;Tan, Nigel;Bhowmick, Sanjukta;Taufer, Michela
• 通讯作者: Taufer, Michela

A Survey of Graph Comparison Methods with Applications to Nondeterminism in High-Performance Computing

• DOI: 10.1177/10943420231166610
• 发表时间: 2023-04-05
• 期刊: INTERNATIONAL JOURNAL OF HIGH PERFORMANCE COMPUTING APPLICATIONS
• 影响因子: 3.1
• 作者: Bhowmick，Sanjukta;Bell，Patrick;Taufer，Michela
• 通讯作者: Taufer，Michela

Identifying Degree and Sources of Non-Determinism in MPI Applications Via Graph Kernels

通过图内核识别 MPI 应用中非确定性的程度和来源

• DOI: 10.1109/tpds.2021.3081530
• 发表时间: 2021
• 期刊: IEEE Transactions on Parallel and Distributed Systems
• 影响因子: 5.3
• 作者: Chapp, Dylan;Tan, Nigel;Bhowmick, Sanjukta;Taufer, Michela
• 通讯作者: Taufer, Michela